I had now my Zimbra Desktop 2.0.1 crashing twice and giving me this error:

2011-04-06 23:47:07,163 FATAL [sync-manager-init] [] system - Unable to commit database transaction.  Forcing server to abort.
com.zimbra.common.service.ServiceException: system failure: committing database transaction
ExceptionId:sync-manager-init:1302090427162:3c3d85a6c03d400e
Code:service.FAILURE
	at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:248)
	at com.zimbra.cs.db.DbPool$Connection.commit(DbPool.java:125)
	at com.zimbra.cs.mailbox.Mailbox.endTransaction(Mailbox.java:7017)
	at com.zimbra.cs.mailbox.Mailbox.getConfig(Mailbox.java:1277)
	at com.zimbra.cs.mailbox.Mailbox.finishInitialization(Mailbox.java:443)
	at com.zimbra.cs.mailbox.DesktopMailbox.finishInitialization(DesktopMailbox.java:79)
	at com.zimbra.cs.mailbox.SyncMailbox.finishInitialization(SyncMailbox.java:90)
	at com.zimbra.cs.mailbox.DataSourceMailbox.finishInitialization(DataSourceMailbox.java:131)
	at com.zimbra.cs.mailbox.MailboxManager.getMailboxById(MailboxManager.java:483)
	at com.zimbra.cs.mailbox.MailboxManager.getMailboxByAccountId(MailboxManager.java:336)
	at com.zimbra.cs.mailbox.MailboxManager.getMailboxByAccount(MailboxManager.java:272)
	at com.zimbra.cs.mailbox.MailboxManager.getMailboxByAccount(MailboxManager.java:231)
	at com.zimbra.cs.offline.OfflineSyncManager.backgroundInit(OfflineSyncManager.java:728)
	at com.zimbra.cs.offline.OfflineSyncManager.access$400(OfflineSyncManager.java:70)
	at com.zimbra.cs.offline.OfflineSyncManager$1.run(OfflineSyncManager.java:650)
	at java.lang.Thread.run(Thread.java:680)
Caused by: java.sql.SQLException: SQL logic error or missing database
	at org.sqlite.DB.throwex(DB.java:361)
	at org.sqlite.DB.exec(DB.java:76)
	at org.sqlite.Conn.commit(Conn.java:342)
	at com.zimbra.cs.db.DebugConnection.commit(DebugConnection.java:42)
	at com.zimbra.cs.db.RetryConnection.superCommit(RetryConnection.java:120)
	at com.zimbra.cs.db.RetryConnection.access$000(RetryConnection.java:11)
	at com.zimbra.cs.db.RetryConnection$7.execute(RetryConnection.java:112)
	at com.zimbra.cs.db.AbstractRetry.doRetry(AbstractRetry.java:44)
	at com.zimbra.cs.db.RetryConnection.commit(RetryConnection.java:116)
	at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:301)
	at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.commit(PoolingDataSource.java:200)
	at com.zimbra.cs.db.DbPool$Connection.commit(DbPool.java:123)
	... 14 more

The problem: Zimbra Desktop stores some information about mailbox folders and messages in sqlite databases so it can access the messages faster. These databases can sometimes become corrupt.

WARNING: It has worked for me, but you may loose all your data

In the Zimbra Directory where all your user data is, you will find a directory called sqlite, in this directory you will find files like:

mboxgroup1.db
mboxgroup1.db-journal

Do not worry about files ending with .db-journal

For each file ending with .db do the following:

echo .dump | sqlite3 mboxgroup1.db | sqlite3 mboxgroup1.newdb
mv mboxgroup1.newdb mboxgroup1.db

You can do all the files in one go, or do one at a time, and see if the problem is fixed. It is hard from the log to find which file is corrupted, so try your luck.

The .dump command tells sqlite to dump in a txt format the mboxgroup1.db database and then to reload this txt format into the mboxgroup1.newdb database

This effectively repairs and compacts the database.

For a long time in Fiji there were only 3 Autonomous System Number (ASN), Fintel (2941), USP(24390) and a F-Root server(3557) hosted by USP.

These ASN represent network clouds interconnected to each other. The interconnection information is exchanged via the Border Gateway Protocol (BGP). BGP tables define the backbone of the Internet: the tables hold the information on how to reach any IP address from any other IP address. BGP provides you with a complete view of the Internet, and you can choose with some capabilities the way your packets will travel from one destination to the other, but more importantly it allows you to have many paths to the rest of the Internet offering redundancy and reliability. When one BGP link is destroyed, the advertised routes are removed from the peering point and the peers of this peer and so on, which generates a reconfiguration of the Internet. If you have more than one BGP link, the other links gain higher priority and your packets are automatically rerouted via the alternate paths.

Wikipedia defines BGP as "The Border Gateway Protocol (BGP) is the protocol backing the core routing decisions on the Internet. It maintains a table of IP networks or 'prefixes' which designate network reachability among autonomous systems (AS). It is described as a path vector protocol. BGP does not use traditional Interior Gateway Protocol (IGP) metrics, but makes routing decisions based on path, network policies and/or rulesets. For this reason, it is more appropriately termed a reachability protocol rather than routing protocol."

In Fiji, the situation until recently (a month ago) is that FINTEL is connected (via the Southern Cross Cable) to the Internet backbone via BGP and is providing, static routes (non BGP) to the various ISPs (CONNECT, UNWIRED, KIDANET, VODAFONE, DIGICEL). Additionally, USP, the University of the South Pacific, with is main campus in Suva, Fiji, has its own Internet link to AARNET. USP is not connected locally to FINTEL. Internet packets from a student at home, have to go via Australia to reach an online courseware in the University, just across the road. USP also installed a Root Server on its network, this root server is only connected to USP network and therefore does not benefit any other ISP in Fiji.

A month ago, TFL, Telecom Fiji Limited (CONNECT parent company) got access to the Southern Cross Cable and set up its own link to the Internet via the ASN 45349, but as of today, FINTEL, TFL, USP are still not interconnected.

At the moment, none of these entities offer BGP peering, to any organization in Fiji. Why would you want BGP, static routes are more than enough, is the common question when asked about BGP peering in Fiji. BGP is a protocol that allows redundancy and reliability. It self configures depending on network conditions. Having more than one peering point, allows better operation, less downtime, etc... This is very important for the development of the Internet economy in Fiji, opening a wide range of operations and local content.

How can you get therefore reliability, and redundancy using BGP when no local ISPs offers you BGP Peering?

The answer is tunnels. Using the same method to get IPv6 when you local provider does not have native IPv6, you establish an IPv6 tunnel over IPv4 to a remote host. Here the trick is to create a tunnel, to make two peers look like next to each others. You then move IPv4 over IPv4. The whole tunnel is considered as a link (like a phone line, or DSL link), BGP can then run on top of this link, and provide the same capabilities as if the link was a physical link between two modems. You then use at the other end a remote router, which has BGP peering to the rest of the Internet, establish a tunnel from your office router to this router and then BGP peer with it. However you need to be careful that the route advertisement do not overwrite the route that allows the tunnel to function. Therefore on each router, you enter a static route to indicate the other end of the tunnel is not reached via the tunnel but via the local link. Because this route is more specific it will have priority over more generic routes.

It is indeed a cumbersome set up, but it has proven very reliable for our AS55327, it has also reduced drastically our downtime, and network reconfigurations each time one of our ISPs is down, making a breeze to operate a Network Operating Centre in Fiji. Until local ISPs offer BGP peering to finally fulfill the objectives of becoming the Internet hub of the Pacific, development in Internet in Fiji will be very limited. Call Centers, Data Centers, Internet Exchange points, Education (cf lack of peering with USP), local hosting, Content Delivery Network (CDN), will be very difficult to be established without this fundamental block.

In a paper at http://www.caida.org/publications/presentations/2006/wealthofnetworks/ Tom Vest, Internet Economist at CAIDA and OECD correlates the wealth of an economy to the wealth of networks and the number of ASN an economy has.

One of the transition mechanism for IPv6 is 6to4. This is an automatic mechanism which encapsulate IPv6 packets inside IPv4 packets. The packets are sent to relays which are located around the world. A list of these public relays are available at:

http://www.bgpmon.net/6to4.php?week=4

How these relays work?

They use the same technology as for the DNS: anycast. The IPv4 relay is always at 192.88.99.1 on 192.88.99.0/24 network, and this network is advertised via BGP to the rest of the world. It means your 6to4 client will look for the closest (in terms of BGP network) relay. All the relays advertise the same route, therefore your closest router may see a few routes to the various relays in the world and will choose the closest, the same way your client choose the closest DNS root server.

So your 6to4 device will encapsulate an IPv6 packet via IPv4 to the closest relay. This relay will convert it to IPv6 send it and get the answer back. The IPv6 address is well defined and is part of the 2002::/16 network. Similarly the relay will advertise this route on the BGPv6 network, so the packet can come back this way. Or via the closest on IPv6 network 6to4 relay. Communications can be quite asymmetric with 6to4.

Format of an 6to4 IPv6 address
2002:IPv4 address of your 6to4 client:00:MAC-48 derived address of your IPv6 device
16bits:32bits:16bits:64bits=128bits

If you are an ISP get a 6to4 relay in place

It seems a lot of devices offer 6to4 out of the box, and sometimes unknown to the user. Because the list of public relays are limited, it is important that ISPs, looking at offering IPv6, set their own relay. The relay does not need to be public, BGP allows to advertise routes to some networks only. So an ISP could make their relay known only to their customers. This would improve user experience, and also allows the ISP to control the relay. 6to4 is not a bad method, but MTU configuration could be quite a challenge when you don't fully control the network.

Vodafone Fiji Limited has taken a step forward towards the Internet Protocol Version 6 (IPv6) by purchasing gear that supports IPv6.

This was confirmed by Vodafone Information Technology Manager Ateen Kumar who said the transition would bring about changes to how the company operated, however plans to transit were yet to be decided on.

He said once IPv6 was implemented by Vodafone Fiji, they would need to build the knowledge of managing IPv6 addresses and IPv6 IP routing.

He said people in the Pacific were not concerned thus very little action to transit to IPv6 will be taken as a result.

Mr Kumar added that in the Pacific, he saw Internet Service Providers implementing IPv6 after three or more years while some businesses are still not aware of it's importance and benefits.

“Because there is sufficient IPv4 at the this point in time, the issue will arise when more addresses are required. This will leave Internet Service Providers no choice but to implement IPv6,” Mr Kumar said.

He said as for the telecommunications field, implementing IPv6 would bring about major changes which should be managed well since customers would need to support IPv6 on their devices such as broadband modems etc.

People will benefit once IPv6 is implemented as there is an abundant supply of IP addresses to allocate to all customers, devices and servers on the World Wide Web.

The transition from IPv4 to IPv6 will be slow unless there are external driving factors pushing deployment in the Pacific says Digicel Chief Technology Officer Eugene O'Shaughnessy.

He said in the next 6 to 12 months he sees no progress in the transition from IPv4 to IPv6 since more awareness was needed on the benefits and advantages of it.

“Aside from increased address space, not many people in Fiji and the Pacific Islands care about mobility, auto configuration etc. IPv6 will benefit us by increasing space and allow all manner of devices to have public static IP addresses,” Mr O'Shaughnessy said.

He said even though all Digicel's equipment are IPv6 capable they did not have any current plans to transit to IPv6 yet.

Mr O'Shaughnessy said businesses in the Pacific were not aware about the importance of IPv6 and how to adapt to it as a result there will be a very slow transition to IPv6.

Pacific Islands Telecommunications Association is positive that Fiji will follow global trends and transit from IPv4 to IPv6 sooner rather than later.
PITA's manager Fred Christopher says people in the Pacific are well aware and informed on the benefits and advantages of IPv6 especially network operators who need IP addresses.

He said IPv4 addresses are still operating globally and therefore network operators would need to operate both IPv4 and IPv6 if they upgrade to IPv6.

“Tunneling with IPv6 is already happening with a number of operators doing this to get an operating experience with IPv6. There are cost elements to change and with IPv6 now inherent in new equipment manufactured, the issue of cost with equipment may not be significant,” Mr Christopher said.

He said the world has become more closer with the Internet and the main important aspect for people nowadays is access. Mr Christopher said if people did not upgrade to IPv6, the rest of the world will, thus less and less access to the Internet and no access to a number of services provided on the Internet.

In 2003, PITA brought Asia Pacific Network Information Centre (APNIC) to Fiji to raise awareness on IPv4 address depletion, since then there have been workshops and training held at technical level to equip members with necessary knowledge and skills to deploy IPv6.

One year left

We have just passed a milestone, there is only one year left of IPv4 allocations from IANA to the Regional Internet Registries: APNIC, ARIN, RIPE, LACNIC, AfriNIC. The Regional Internet Registries RIR will need a few more months to allocate their available resources to ISPs, and some more months from the ISPs to the end users. What is left in the IPv4 pool is a bit dirty as these are addresses that have been in use, against recommendations, on internal networks. There has been some studies to see how dirty this space is. It will not mean the end of the Internet in one year, but certainly users will select their ISPs/Collocation Centre in function of which one can provide them with address space. There could be address space trading and any kind of other oddities.

What has to be done?

Migrate to IPv6 is the solution. Adding NATs (using private IP space behind one public IP address), is like wasting money in an old car. There is a lot of misconception out there about the difficulty of implementing IPv6, but it is damn easy, on the client side that is. Enabling your client to surf the Internet on the IPv6 takes about 5mn. You configure your router internal interface with an IPv6 address and let the router advertise the new network. No need of DHCP, static IP, etc... The internal computers will pick it up, self configure and start to go IPv6. Add a filtering rule on the router to avoid the Internet to connect to your internal computers and you are as safe as with a NAT.

Where is the difficulty?

The difficulty is in configuring your servers to serve to IPv6 clients. I recommend to disable IPv6 on your servers, have a bit of fun with your IPv6 clients on your internal network, and then enable one by one your servers to work over IPv6. You need to make sure your software will listen on IPv6 addresses and you don't have any special code/script/database that won't understand the new IPv6 format. If your engineers are using IPv6 on their desktop they will become more familiar with it, and think about the implementation of it in code, all naturally.
So yes it is moving in two steps: clients first, servers later.!

Why IPv6 did not pick up early

IPv6 has been here for the last 10 years, but I think the trigger on the adoption of IPv6 was IETF 71, March 9-14, 2008; Philadelphia, PA, USA. At this meeting it was decided to do a one hour long IPv4 outage. Participants will have no other choice than to use IPv6 strictly to surf the Internet. They would be able to see what works, what does not work, what can be fixed with workarounds, and what's left to do... It was at that time that ICANN put the IPv6 glue in the root servers, so it was now possible to query the DNS with IPv6 only. You can retrieve an IPv6 address using the IPv4 DNS stack, it is not an issue, and as machines are dual stack (IPv4 and IPv6) they can use either stack to do their DNS queries but IPv6 has to be able to stand on its own ground. The other event was Google provided http://ipv6.google.com for people to have a site to go to and test their configuration. Since then Google has migrated all their services to IPv6 (including YouTube!). Netflix has done the same and many others are following. It means once you enable IPv6 on your network, you see a lot of IPv6 traffic. First all your clients will self configure to IPv6 (see above) second there are many popular IPv6 sites out there. It does not grow slowly.

Where are we in Fiji and the rest of the world.

We went around and asked the various stakeholders what were their plans for IPv6. So far the response have been interesting but not surprising, everyone says they have to, but no one has a definitive, well thought, plan. It is a bit of let's see attitude. We are still waiting for answers from FINTEL, Vodafone and Digicel. I think in the coming year we will see an exponential IPv6 uptake as the word spreads that it is easy to enable your client network to IPv6. Europe and Asia are quite in advance on IPv6 deployment, but the USA lags behind, and as the USA and the Silicon Valley leads in IT innovation, until they move to IPv6 there will be still this wait and see attitude. Once the Silicon valley goes IPv6, expect a snow ball effect. There is a lot of noise on IPv6 there already like the IPv6 panel by the San Francisco bay Area Chapter of the Internet Society or the Google IPv6 implementators conference

The coming year will be very quite interesting...

This week we migrated our Zimbra mailsever to send and receive email over IPv6.

While Zimbra does not officialy supports IPv6, the underlying component to send and receive email, postfix, supports IPv6 for a long time. The process is therefore to modify the postfix config to enable IPv6. However Zimbra rewrites the various components configuration files, therefore you cannot edit them directly. You edit /opt/zimbra/conf/zmmta.cf and add the line in the mta section

POSTCONF inet_protocols all

like in the example below:

...
  POSTCONF virtual_transport LOCAL postfix_virtual_transport
  POSTCONF inet_protocols all
RESTART mta
...

As s zimbra user, restart the mta:

zmmtactl restart

Additionally, we modified the MTA mynetworks parameter. This cannot be edited directly from the GUI. The GUI checks that you enter a valid IP address, but checks only for IPv4.

You can do that directly as the zimbra user:

zmprov modifyserver seine.avonsys.com zimbraMtaMyNetworks '127.0.0.0/8 x.x.x.x/x [::1]/128 [xxxx:xxxx:xxxx::x]/48'

Where [xxxx:xxxx:xxxx::x]/48 is your IPv6 network.

All is explained on Zimbra forum and on Zimbra Bugzilla

We then modified our MX record to point to our machine which will have both a A (IPv4) and AAAA (IPv6) record. Dual stack clients usually prefers IPv6 over IPv4.

The last part was to get the rDNS set up. Mail servers are using the reverse DNS to check there are linkage between the IP that sends email, the machine it claims to be and the domains used in the email itself. Basically, when a server receive a connection, the sender will identify itself via the EHLO or HELO command: 'HELO seine.avonsys.com'. The sender would be connecting from a specific IP: 2001:df0:67::129, the receiver check via the rDNS what hostname is associated with this IP.

dig -x 2001:df0:67::129
;; QUESTION SECTION:
;9.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.7.6.0.0.0.f.d.0.1.0.0.2.ip6.arpa. IN PTR

;; ANSWER SECTION:
9.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.7.6.0.0.0.f.d.0.1.0.0.2.ip6.arpa. 604800 IN PTR	seine.avonsys.com.

You can see the format of the rDNS for IPv6. You reverse the full IPv6 address (add the 0 where they should be):

2001:0df0:0067:0000:0000:0000:0000:0129

Now the issue, is that there are very little IPv6 enabled DNSBL. A DNS Blocking List, is a DNS which answers in codes in the range 127.0.0.x for any IP which is deemed unsafe to receive email from. First few DNSBL operators have IPv6, second, bind the popular DNS software is rarely used for such task as it is not optimized for this specific problem, instead people use rbldnsd which till recently was not IPv6 compatible, finally the sheer size of the database to store all these invalid IPs may be too much to handle (There are techniques to not consider all IPv6 addresses possible).

One such IPv6 DNSBL (and may be the first one) is VIRBL, at the moment we have no information on how to make it work with Zimbra for IPv6, but stay tuned.

A study by RIPE shows the level of SPAM on IPv6, but human sighting of SPAM on IPv6 tends to show that the SPAM does not originate from an IPv6 host, but from an IPv4 machine, and then relayed by a dual stack IPv4/IPv6 email server. Surely, things will change.

Overall, it is easy to configure a mail server on IPv6, relatively safe as the level of SPAM is not at the levels of SPAM on IPv4, and the second line of defense which are content based filters are not affected by the connected IP.

Now, we will gain experience on the practice of having SMTP over IPv6.

In May this year Avonsys was amongst other Information Technology companies that attended the Web 2.0 Expo which was held in San Francisco.

The Web 2.0 Expo is an annual event which gathers web developers, innovators, companies and individuals who are the next generation on the web under one roof to showcase their products.

The Web 2.0 Expo provides a platform that enables developers or IT specialists to connect with other individuals who are also focusing on Software as a Service and web-based technologies.

Two senior executives from Avonsys attended the Web 2.0 Expo bringing back many exciting stories of the events that unfolded during the three busy days of the expo.

An executive said this year’s Web 2.0 Expo emphasized on the lack of specialized services that are being offered in the Pacific.

He said most IT-oriented companies in the Pacific are offering standard and not specialized services compared to Avonsys.

Attending the Web 2.0 Expo annually allows Avonsys to interact with their target market and also gives the company a chance to profile itself and its services on the West Coast of the United States of America.

The executive said one of their main activities during the Web 2.0 Expo was to explain to audiences the key services carried out by Avonsys.

This meant interacting with people who visited the Avonsys booth as well as other exhibitors and attendees in general, this year's Web 2.0 Expo saw a large amount of interest from providers of cloud computing infrastructure.

This indicated the growing popularity of cloud computing as a platform to offer services, replacing traditional on-premises software installation.

The Web 2.0 Expo 2010 was a success for all.

While the deployment of IPv6 is relatively easy it is still limited in the enterprise to the capability of the hardware. For instance lot of enterprise needs system with redundancy and failover mode. For instance Cisco ASA provides a firewall solution with hot standby capability. You place two boxes next to each others. the configuration of one will be sync to the other box. One box will be declared a Primary and the other Secondary. Once the primary is not any more detected by the secondary, the secondary picks up the IPs of the interfaces of the primary and act exactly like the primary. This is all well done using IPv4 addresses but the capability is not available in IPv6 until now with version 8.2 of the ASA OS.

There are still a lot of IPv4 functionalities that do not exist in IPv6 for lack of implementation. It means for many organisations, they have to move cautiously, or be aware they cannot offer the same level of reliability till they are able to upgrade their software. Until we can do in IPv6 what we do in IPv4 there will be still challenges to move to IPv6, and corporations don't like to find these issues the hard way.