We moved this page to our Documentation Portal. You can find the latest updates here. |
Issue
If you see the following in a VM startup failure log:
onappstore activate failed for vdisk xxxxxxxxxx with error map: [] and optional error: assertDictsMatch
or:
Dictionaries do not match for key snapshots
Resolution
If you look at the 2 dictionaries in the log:
Fatal: OnApp::Actions::Fatal Storage API Call failed: {"result"=>"FAILURE", "error"=>"onappstore activate failed for vdisk geip2ck3ajd65t with error map: [] and optional error: assertDictsMatch - Dictionaries {'frontend': [u'0.0.0.0'], 'snapshots': [u'i89at0sjo36b4x'], 'datastore': '1orn8jk2sf0cvb', 'numkeys': 855732L, 'port': 47434, 'size': 107374182400L, 'sector_size': 65536, 'insyncstatus': 2, 'uuid': u'geip2ck3ajd65t', 'utilization': 855732, 'st_size': 256, 'transaction_id': 0, 'status': 0, 'description': u'e3q1tf44ax48yi', 'replicas': 2, 'seqno': 436357824L, 'membership': [u'2239970245', u'780556693'], 'members': '2239970245', 'utilization_percent': 52L, 'name': u'e3q1tf44ax48yi', 'sectors': 1638400L, 'st_mems': 1, 'snapshot': 0, 'st_mem': 0} and {'frontend': [u'0.0.0.0'], 'snapshots': [], 'datastore': '1orn8jk2sf0cvb', 'numkeys': 855734L, 'port': 22008, 'size': 107374182400L, 'sector_size': 65536, 'insyncstatus': 2, 'uuid': u'geip2ck3ajd65t', 'utilization': 855734, 'st_size': 256, 'transaction_id': 0, 'status': 0, 'description': u'e3q1tf44ax48yi', 'replicas': 2, 'seqno': 436357824L, 'membership': [u'2239970245', u'780556693'], 'members': '780556693', 'utilization_percent': 52L, 'name': u'e3q1tf44ax48yi', 'sectors': 1638400L, 'st_mems': 1, 'snapshot': 0, 'st_mem': 0} do not match for key snapshots"}
One reports a snapshot child:
'snapshots': [u'i89at0sjo36b4x']
The other reports:
'snapshots': []
So, the node 2239970245 is reporting a snap which needs to be cleaned up:
[root@x.x.x.x ~]# onappstore nodeinfo uuid=2239970245
result=SUCCESS lastping=1384733762 status=1 datastoremember=1orn8jk2sf0cvb hostid=6 vendor=ATA uuid=2239970245 hbtl=0:0:0:0 freespace=358270976 nodestatus=0 utilization=15 ipaddr=10.200.6.1 performance=1 vdisks=jftnvy0ez34md1,69x4nbszck8q0u,xu9a7lv0h1bcrj,s6ktgj5ay84qdu,h1smg7cd93bp0v,geip2ck3ajd65t,s0gbjq1pawykc5 version=v1.4 statusinfo=Active size_bytes=750156374016 allocated_space=243739394048 model=WDC_WD7500BPKT-7 revision=01.0 serial=WD-WXD1E33SXJV9 size=1465149168
Logon to backend node via telnet and look for any files matching that snapshot ID:
/ # find /DB -name "*i89at0sjo36b4x*"
/ #
none
So, a stale entry is in the redisDB on that node only, and a reset of the DB should resolve this issue:
/ # ps w| grep group
1635 0 10372 S /usr/pythoncontroller/python /usr/pythoncontroller/groupmon -i eth0 -B -c 224.3.28.2
1642 0 10372 S /usr/pythoncontroller/python /usr/pythoncontroller/groupmon -i eth0 -B -c 224.3.28.2
1643 0 10372 S /usr/pythoncontroller/python /usr/pythoncontroller/groupmon -i eth0 -B -c 224.3.28.2
1645 0 10372 S /usr/pythoncontroller/python /usr/pythoncontroller/groupmon -i eth0 -B -c 224.3.28.2
1646 0 10372 S /usr/pythoncontroller/python /usr/pythoncontroller/groupmon -i eth0 -B -c 224.3.28.2
1647 0 10372 S /usr/pythoncontroller/python /usr/pythoncontroller/groupmon -i eth0 -B -c 224.3.28.2
5177 0 10372 S /usr/pythoncontroller/python /usr/pythoncontroller/groupmon -i eth0 -B -c 224.3.28.2
10131 0 1864 S grep group
/ # killall groupmon
/ # killall redis-server
/ #/ # /usr/pythoncontroller/groupmon -i eth0 -B -c 224.3.28.2
/ #
Then, from the HV directly:
[root@x.x.x.x ~]# onappstore diskinfo uuid=geip2ck3ajd65tresult=SUCCESS status=0 sector_size=65536 insyncstatus=1 memberinfo={u'780556693':{u'status':0,u'frontend':[u'0.0.0.0'],u'seqno':436357824,u'membership_gen_count':5,u'st_mem':0,u'numkeys':855734,u'port':22008},u'2239970245':{u'status':0,u'frontend':[u'0.0.0.0'],u'seqno':436357824,u'membership_gen_count':5,u'st_mem':0,u'numkeys':855732,u'port':47434}} description=e3q1tf44ax48yi sectors=1638400 replicas=2 utilization=855734 st_mems=1 snapshots=[] membership=[u'2239970245', u'780556693'] st_size=256 members=780556693,2239970245 Datastore=1orn8jk2sf0cvb utilization_percent=52 datastore=1orn8jk2sf0cvb size=107374182400 uuid=geip2ck3ajd65t snapshot=0 transaction_id=0 name=e3q1tf44ax48yi
Let's check if it can online and offline ok:
[root@x.x.x.x ~]# onappstore getid
ipaddr=x.x.x.x result=SUCCESS uuid=194752535
[root@x.x.x.x ~]# onappstore online uuid=geip2ck3ajd65t frontend_uuid=194752535
result=SUCCESS
[root@x.x.x.x ~]# onappstore offline uuid=geip2ck3ajd65t frontend_uuid=194752535
result=SUCCESS