Suramya's Blog : Welcome to my crazy life…

February 22, 2021

Should software be “classified” if it is used by government agencies?

Filed under: My Thoughts,Techie Stuff — Suramya @ 10:25 PM

Software should not be classified when used in government agencies except in some very rare exceptional cases. For example, software used to launch nuclear missiles and ensure they hit their targets might be an exception however, these should be an exception rather than the rule in my opinion. The reason I say that is because when a software is classified it means that a limited no of people are using it and that means only a limited no of developers are working on it as any given time. Plus, when something is classified it can not be security tested by external entities without having to sign multiple Non-Disclosure Agreements (NDA’s) and other similar hoops whereas regular software is used and tested by lots of people like corporate’s who test the solution before they implement it in their environment. This gives us multiple groups of people testing the software in various ways making it easier to identify security issues making it more secure.

Secondly, making classified versions of existing software doesn’t seem like a good use of resources to me. Why spend extra money creating a classified ‘Word’ when the commercial version (with some modification) would work perfectly well?

Coming to open source, I don’t think government should be prohibited from using Open Source. There are two ways government’s can be blocked from using opensource software. Firstly, the software license might prohibit government use. If this is the case, then the license no longer meets the requirement to be called open source since the Open Source Definition specifies that Open Source licenses may not discriminate against persons or groups. In-fact there was an attempt back in 2006 by GPU which is a Gnutella client to create a license that disallowed the use of their software by the military. It didn’t go anywhere because of strong opposition by the Open Source Initiative (OSI) to calling the license open source. Second, The government might not allow its use due to internal policy: This is primarily because they are worried about one or all of the following: Security, Cost, or licensing.

There is a misconception that opensource software is less secure than commercial software and even though majority of the servers worldwide run on opensource software (Linux, Apache etc) the detractors still persist. In every large company I have worked with so far, whenever I try promoting opensource software this point invariably comes up and I have to spend time explaining why this is not the case.

Another reason why companies might not allow OSS is because of the license the software might be under. If the software is licensed under the GPL they need to be extra careful when using it within their setup because the GPL is what is known as a viral license. It requires the source code of the entire “derived work” – i.e. the combination of GPL code and proprietary code – to be made available under the terms of the GPL publicly. Due to this most companies do not allow software licensed under the GPL to be used inside them. (Libraries licensed under GPL are fine because they don’t trigger the ‘viral’ clause). However, there are other licenses that do not have this clause and can be safely used.

Ultimately, I feel it is better if they use opensource software rather than pay ridiculous amounts of money as software licensing fees.

February 21, 2021

All Your Base Are Belong To Us: Turns 20

Filed under: Humor,My Life,My Thoughts — Suramya @ 12:05 AM

‘All Your Base Are Belong To Us’ is a phrase that is very familiar to anyone who was active on the internet in 2000’s as this was pretty much the first meme which took the entire net by storm. I remember someone posting a giant version of it in our college dorm windows using A4 paper. Unfortunately I don’t have a pic of it, but there were plenty of other places this popped up over the weeks really confusing a lot of people.

The first version showed up online on February 16, 2001 when a robo-voiced music video went live at Newgrounds.com. The video was a capture from a 90’s video game called Zero Wing with some hilariously bad English translation from the original Japanese. It contained phrases like “How are you gentlemen!!”, “Somebody set up us the bomb” and the infamous “All Your Base Are Belong To Us!”. The video presented the original Sega Genesis graphics, dubbed over with monotone, machine-generated speech reading each phrase. The video is shown below, click on play to view the original video with the awesome soundtrack:


All Your Base Are Belong to Us

The transcript of the video is as follows:

Captain: What happen ?
Mechanic: Somebody set up us the bomb.
Operator: We get signal.
Captain: What !
Operator: Main screen turn on.
Captain: It’s you !!
CATS: How are you gentlemen !!
CATS: All your base are belong to us.
CATS: You are on the way to destruction.
Captain: What you say !!
CATS: You have no chance to survive make your time.
CATS: Ha ha ha ha …
Operator: Captain !!
Captain: Take off every ‘ZIG’!!
Captain: You know what you doing.
Captain: Move ‘ZIG’.
Captain: For great justice.

After the original video from the game it moves on to show the phrase photoshopped into all sorts of images and photos from actual locations where it was inserted really confusing most folks because they had no idea what it meant.

It was extremely silly but great fun. Watching it again has really brought back memories of the great times from college. Here’s to the next 20 years, ’cause remember “All Your Base Are Belong To Us!”

– Suramya

February 20, 2021

Fixing boinc (code=exited, status=108) error

Filed under: Computer Tips,Knowledgebase,Linux/Unix Related — Suramya @ 2:01 AM

Earlier today I noticed that my CPU was not as active as usual and the boinc (World Community Grid) processes were no longer active on my computer. This has happened in the past when the client crashed so I restarted the client using the following command as usual:

/etc/init.d/boinc-client restart

Unfortunately, that didn’t resolve the problem and I thought that it could be because of the recent OS update that I did to my Debian system. In the past there have been rare cases when libraries were updated that some programs act strangely till the computer is rebooted, so I restarted the machine expecting to see the process start up without issues. Sadly, that didn’t happen so I had to debug the problem and I tried all sorts of things to resolve.

First, I tried starting the program manually as the root user and that worked. So I knew it was something to do with the startup script. Then I searched for and removed all the lock files in the boinc and the boinc-client directory. That should have resolved the problem but it didn’t and then I tried running the status command which gave the following output:

root@StarKnight:/var/lib/boinc-client# /etc/init.d/boinc-client status
boinc-client.service – Berkeley Open Infrastructure Network Computing Client
Loaded: loaded (/lib/systemd/system/boinc-client.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2021-02-20 01:26:50 IST; 9s ago
Docs: man:boinc(1)
Process: 7420 ExecStart=/usr/bin/boinc (code=exited, status=108)
Process: 7455 ExecStopPost=/bin/rm -f lockfile (code=exited, status=0/SUCCESS)
Main PID: 7420 (code=exited, status=108)
CPU: 19ms

Feb 20 01:26:40 StarKnight systemd[1]: Started Berkeley Open Infrastructure Network Computing Client.
Feb 20 01:26:50 StarKnight boinc[7420]: 20-Feb-2021 01:26:50 Another instance of BOINC is running.
Feb 20 01:26:50 StarKnight systemd[1]: boinc-client.service: Main process exited, code=exited, status=108/n/a
Feb 20 01:26:50 StarKnight systemd[1]: boinc-client.service: Failed with result ‘exit-code’.

This meant that the system thought that another instance of the software was running but that wasn’t the case as I verified it using ps. A search for the status=108 code on the internet returned a few results but nothing that resolved my problem. One user who faced this issue resolved it by uninstalling everything and installing back but that wasn’t a step I wanted to take without trying everything else first so I kept researching. Then I saw a post where a user was facing the same issue after they had moved the data directory to another partition and symlinked it to the original location. I had done the same thing a few weeks ago so I moved the directory back to it’s original location but that didn’t resolve anything either.

Then I thought about checking the file ownerships of the directory and they were owned by my user (suramya) and a post on the internet said that they should be owned by root. I checked on my laptop as I have the same setup there and found that the directories were owned by the ‘boinc‘ on the laptop. Then I remembered changing the ownership of all files in one of my drive partitions last night to suramya. What I didn’t realize at that time was that the boinc-client directory was also located on that partition (after I had moved it there to recover space on my root partition).

I immediately changed the ownership of both directories back to boinc:boinc using the following command

chown boinc:boinc /var/lib/boinc* -R

Then I restarted the daemon and that fixed the problem. I then moved the directory back to it’s original location (on the other partition), symlinked it to the original location and the software still worked after I restarted the process.

I am documenting this in case others hit the same issue.

– Suramya

February 7, 2021

Travel in 2021: 3294kms driven as of date

Filed under: My Life,Travel/Trips — Suramya @ 8:25 PM

I complained a lot about not being able to travel in 2020, mostly to my wife who suffered through my complaints heroically. So the universe decided that I would get to travel a lot in 2021 as compensation. Jan 2021 has been a marathon session of driving all over south India, primarily because of Covid as we only wanted to go places within drivable distance from Bangalore as we normally would try to drive directly to the location without having to stop on the way. Plus we also ensured that we were only going to places with very limited crowds and mostly ended up staying in places where we were the only people (apart from the staff).

The first trip of the year was to Dindigul where we had gone for New Years. It was a lot of fun and we enjoyed it quite a bit. The second trip was to Kannur, Kerala (not to be confused with Coonoor, Tamil Nadu), which we did with Jani’s friends from IBM with whom we had been talking about going on a trip for over 4 years (almost 5). The drive to the place was quite nice and we made good time. I had planned on writing a trip summary as I normally do but didn’t get the time as I was swamped with classes and assignments. In any case, Jani has now started writing about all the trips so that reduces the pressure on me :). Her writeup on the trip is available here.

The next trip was to Kotagiri, Tamil Nadu with Anil, Manisha, Suprita , Vinit and the kids. We drove down from Bangalore to Kotagiri but thankfully I wasn’t driving because by this time I had already driven almost 1.8k kms in 2021. Anil, Manisha & Vinit drove for the trip and I got to relax and spend time with the kids for a bit. A big surprise was that I bumped into Gaurang in Ooty, as he had driven down for some work. We both stay in Bangalore but hadn’t seen each other for months because of Covid. So he came over and we spent some time together there and then he drove back with us which gave me a chance to spend some quality time with him during the drive.

Finally the last trip of the month was to visit Jani’s hometown, Colachel in Kanyakumari District for some work (plus spend time with family). It was a 14 hour+ drive from Bangalore and initially I had planned to do it in one shot. However, then we decided to break it up into two with a halt in Dindigul which is almost exactly half-way between the start and finish. This was the first time we were stopping on the way during a trip but in hindsight it was a great idea. It allowed me to take a break and rest as I was also up on night for my classes. If we hadn’t stopped on the way I would have been exhausted by the time we reached and this way I got to rest on the way. The drive down was fantastic, and I really loved the NH44. The road was well maintained and an absolute pleasure to drive. I had to consciously restrict myself from driving too fast and I really enjoyed it. The traffic also was minimal so I could really relax and enjoy the drive.

In all I have driven 3294 kms so far in 2021 and now that I am home I absolutely refuse to drive anywhere else for the month of Feb. We will see how I feel in March, but most probably we won’t be going anywhere till at-least the end of March as we both have exams in March. But based on our past experience, I will not say that we absolutely will not travel anywhere for the next two months.

I am eagerly waiting for the Covid vaccine to be generally available so that we don’t have to worry about Covid when traveling. Looking at the current status I have a feeling that it will be Q3, 2021 by the time we get vaccinated. Hopefully by the end of the year things might be back to normal so here’s hoping for that.

Well this is all for now. Will write more later, now that I am back home and not driving all over south India.

– Suramya

January 11, 2021

Do young people not care about privacy because they grew up sharing personal information?

Filed under: My Thoughts,Techie Stuff — Suramya @ 1:03 PM

I don’t agree with statement, though there are many proponents to abolishing privacy online. Unfortunately, over the years we have been trained to give personal information in order to use services online and this is based on the principle that our data has no value and if we have nothing to hide we should be ok to share data online. This is a fallacy.

Having privacy is essential to us as humans because it gives us a judgment free space where we can be ourselves without worrying about what others might think. This allows us to explore unpopular ideas without worry or fear. Plus it is not just people who have unpopular ideas that need privacy. There is a lot of stuff that I wouldn’t want to share with everyone even if it’s nothing illegal or unpopular such as details about my health or personal finances. Privacy doesn’t mean that we don’t want to share information, it just means that I get to choose who has access to data about me.

In certain scenarios privacy helps protect us physically from stalkers or people who mean us harm, think fanatic fans or people fleeing an abusive relationship. Such people would not want their physical location broadcast to the world. There was a case a few months ago where a fan used a photo published by a star to locate her apartment and assaulted her at her apartment. This was a breach of privacy of the star and caused major distress to her. Now imagine if we could immediately find where anyone in the world is located at a given time, this will allow us to determine so much other data about that person such as their health information (if they are visiting a cardiac doctor every week, there is a high probability they are suffering from a hear problem). A few years ago Uber analyzed data from the rides people were taking using it’s service and used that to figure out who was having an affair with whom based on the rides they took and the location they were dropped. It was quite a scandal when it came out and now imagine someone taking this information and blackmailing people.

With the amount of information that we are giving to websites and companies both voluntarily and involuntarily privacy is becoming harder to maintain but that doesn’t mean that we give up and let companies do whatever they want with our data. If we do that then be prepared to have every aspect of your life dissected and analyzed for profit.

Too many people state that they have nothing to hide and have no problems with having their information public. I challenge them to stay in a house completely made of glass (including the bathrooms) and have a bot that publishes all emails/messages/transcripts from their calls publicly. I can bet there won’t be any takers. as everyone has something they wouldn’t want to be public knowledge.

– Suramya

January 10, 2021

What are the ethical obligations of a Computer professional?

Filed under: My Thoughts — Suramya @ 11:58 PM

This is a question that is getting a lot of attention right now. A lot of people say that Technologists shouldn’t be political or worry about how their tech being used but I believe that it is wrong. This school of thought comes from when historically people working on computers were not impacting any real-world events/consequences. E.g. if a computer crashed or was hacked it might not have a life & death impact. Now with everything connected to each other including devices that effect the physical world that is not the case. So if there is a major flaw in a control system of a car that allows it to be hacked then it can be used to crash a car or stop it in the middle of the road causing a pileup. If there is a vulnerability found in a pace maker then it can be used to kill people.

Due to this all our work needs to take in account all three scenarios. We can’t just create a system that causes extensive harm and claim that it was done because “This is what I was told to do” is not a valid justification for doing something that is used to harm people/communities. Some claim that our job is to help our companies & clients make money and leave the ethics at home but it is not the correct way to look at things.

To take an example, what if I develop hacks that allowed governments to spy on terrorists undetected by monitoring their phones & computers? That seems like a win-win for all correct?I am helping stop terrorists and keeping the world safe. What else do we need? Now what if those same hacks were repurposed by repressive regimes to spy on their dissidents? It shows that everything we do and work on, has consequences some of which are intended & some are not.

In another example, if I figure out a way to remotely identify anyone even when they are masked, then before I release the software I also need to think of how else the software might be used. Will it be used to target protestors or political dissidents? We need to figure out what other uses it might have and then take a call.

Unfortunately there is no clear answer or a checklist that we can follow to make the correct decision. End of the day we need to make a decision and then live with the consequences.

– Suramya

January 9, 2021

Online Afterlives: Chatting with the dead

Filed under: My Thoughts — Suramya @ 2:56 PM

Dealing with death is something that everyone struggles with and with the digital aspects of life becoming more and more prevalent there are many ways folks try to keep the memory of their loved ones accessible. There are options in major social media sites to memorialize an account after the owner passes away, others use personal websites to memorialize their loved ones. With advances in technology there is now a new way to remember your loved ones, using AI and machine learning there are companies that allow you to ‘chat’ with your loved ones even after they have died. Basically these sites train a machine learning model using existing communication, emails, chats, postings etc to give you the impression that you are chatting with a dead person.

In theory this is very interesting and I like the use of technology to ease the sense of loss from a death. However, my concern is that this can quickly become a crutch and for people who are having a hard time letting go, this can make things even more complicated. In the end this is a chat-bot pretending to be a person, although to be fair the bot is explicit in telling people that it is a bot. For example when asked where they were the bot responded: “As a bot I suppose I exist somewhere on a computer server in San Francisco. And also, I suppose, in the minds of people who chat with me.”

Overall I am not sure how I feel about the tech. It is both good and creepy at the same time. Extrapolating into the future we can see that soon it would be possible to create a virtual reality (VR) representation of a person that can interact with people after the original person dies. All you would need is data and enough processing power to create a model of how a person behaves. There was a TV show that I saw a while ago where dead people lived on as online avatars and would still interact with their loved ones but I can’t remember the name but it was an interesting concept.

What do you think?

Source: Popsci.com: Old text messages are letting people chat with the dead

– Suramya

January 8, 2021

Idiot threatens to kill co-worker because his friend request wasn’t accepted

Filed under: My Thoughts — Suramya @ 4:52 PM

A while ago I had posted about the most bizarre reaction I had seen from someone whose friend request I had not responded to quickly (I don’t check FB very often). I am happy to say that it was nothing like the reaction this person from North Dakota, US got. Apparently 29 year old Caleb Burczyk decided that being friends with his ex-coworker was so important that Caleb threatened to kill them if his request wasn’t accepted. To top things off, this moron actually went over to their house and kicked in the front door while getting caught on camera doing so just because he couldn’t take the fact that his friend request was rejected ensuring that he has a felony assault entry in his record.

Caleb Burczyk, 29, pleaded not guilty to felony charges of burglary and terrorizing filed in Williams County District Court Tuesday, Dec. 29. Burczyk’s attorney Jeff Nehring declined to comment on the case.

Police say Burczyk started sending aggressive Facebook messages to his ex-coworker on Dec. 24, according to an affidavit of probable cause. He threatened his ex-coworker’s life and warned him that he was going to “come at” him if he did not accept his Facebook friend request, the affidavit stated.

“Accept my friend request or I’m going to murder you,” Burczyk wrote in a message to his ex-coworker, according to the affidavit.

I can understand being slightly upset when someone doesn’t want to be friends with you but this is extreme and the guy should be in jail as he is not stable and could have caused a lot of harm. These are the kind of entitled morons who need to be taught that the world doesn’t revolve around them and they need to accept that people don’t want to associate with them. I don’t blame the co-worker for not wanting to be friends if this is how he behaved. I am sure he was doing the same thing in real life as well.

Source: PSA: If Someone Doesn’t Accept Your Friend Request, Do Not Threaten To Kill Them And Kick In Their Front Door

– Suramya

January 7, 2021

Welcoming 2021 in style at Rajakkad Estate, Dindigul

Filed under: My Life,Travel/Trips — Suramya @ 5:06 AM

One of the biggest things that I missed in 2020 was traveling. Usually we travel to multiple places over the year but in 2020 it wasn’t the case due to Covid and we barely traveled anywhere. So, we wanted to do a trip for new years because we were getting stir crazy and we wanted to celebrate with a small group of close friends without having to interact with unknown folks. After a bit of research we decided to head down to Rajakkad Estate, Dindigul. This is an 18th century palace that was transported from Kerala by breaking it down to 35000 pieces and re-assembled in Dindigul in the middle of an 80 acre estate. The reviews of the place were amazing so we booked the place and prepared for the journey. Due to Covid we had to register for an e-pass as we were crossing into Tamil Nadu and the pass was auto-approved. Interestingly, there was no check for the epass but I don’t recommend traveling without it because if you don’t have the pass then you are sent back.

The trip started early morning (5:30am) on the 31st with me, Jani, Ayush & Akanksha in my car and Shashank on his bike. Just as we left home it started raining and it continued to rain throughout the trip. Due to which we had to drive at a slower speed and we also ended up stopping a few times due to the heavy rain so that Shashank could take a break from driving in the rain. Thus the trip which was estimated to take ~7 hours took us over 9.5 hours to complete. The last part of the drive was amazing with spectacular views but I didn’t get to enjoy them much as I had to focus on the road and I didn’t want to drive off the cliff while admiring the view. 😉

We finally reached the estate around 3pm and found that we 5 were the only guests over there for the duration, so we got to select our rooms. Me and Jani selected a corner room with spectacular views of the forest. The host Robesh walked us through the place and once we settled in we immediately requested for lunch as breakfast was a long time ago and this time we couldn’t stop for snacks as we usually do due to the rains. Lunch was quite good and every item in the menu was locally sourced from the estate’s own farm. Can’t comment about the non-veg dishes but the veg options were quite tasty and healthy. I do recommend that you stick with the south-Indian food options (they do make continental food but we didn’t try any) as the cooks are not that great with north Indian dishes (especially chappati’s). Other than that one dish most of the dishes were quite good. I even enjoyed the banana flower vegetable which is something I have never liked.


Hogging on great food

After lunch we thought about exploring the surrounding area a bit but it was still raining so we just relaxed in the sitting area and played a bunch of board games (they have a good collection). It had been a while since we all got together so it was good to catch-up with each other’s lives. Thanks to the rain the temperature dropped quite significantly and Jani was quite thankful for her electric heated jacket. We asked if we could get a heater setup in the area but apparently there was no power outlet (15A) that could take the load of the heater so after braving the cold for a bit we moved the gathering to the room where it was a lot more warmer after we closed the windows and the door. (If you are someone who doesn’t like the cold make sure you pack warm clothes as it gets quite cold in the evening as the place is on a hill)


The Central courtyard of the palace

We had initially planned to ring in 2021 with a bonfire, but thanks to the rain we thought that it wouldn’t be possible. However, the staff went out of their way to get a bonfire started for us and we got to sit outside next to the fire to welcome 2021. It was completely unexpected and a very pleasant surprise. After enjoying the fire for a bit we moved back to the room due to the cold and spent another couple of hours just chilling. We couldn’t stay up too late because all of us were tired as we had an early start so we crashed.


Welcoming the New Year with a bonfire


Cheers to having a fantastic 2021

The next day, I woke up at 8am for some reason and then couldn’t go back to sleep, so I spent a very pleasant morning walking around the surrounding woods and exploring the lovely garden. Once everyone else woke up we had a great breakfast with fresh juice, homemade bread, south Indian dishes and eggs. Thankfully it had stopped raining and the weather was very pleasant so we decided to explore the surrounding area and walked over to the Yoga platform which is built into the side of the hill with a spectacular view of the hills and the sun-rise (if you wake up early enough). We didn’t wake up that early so we just sat there for a bit enjoying the view and listening to Jani talk about all the flowers that she could identify (which were a lot). We then went for an hour long walk in the forest and saw a whole bunch of flora & fauna. The walk helped us build up an appetite and we were reading to do justice to the lunch prepared for us. It was served in an outdoor seating area and we all really enjoyed the food and the location.


Jani and me at the front garden


Freshly plucked tamarind directly from the tree


Group Selfie at the estate


Chilling at the Yoga platform

Post lunch we relaxed for a bit and Robesh suggested we checkout a waterfall nearby so we all drove for about 15 mins to the entrance to the trail and then walked down to the fall. It was a nice walk with a rope bridge on the way that we had to cross. It was amusing to watch some of the folks crossing the bridge (I am not going to name names) as they were quite scared. We did see a couple of giant squirrels on the way along with a whole bunch of birds and that was quite nice. Once we got back we remembered that we had brought badminton racquets so we played for a bit till it became too dark to see, post which we had dinner and another round of games, great conversation and relaxed. There is not a lot to do at this place so be prepared to entertain yourselves. The phone and 4G signal was quite spotty in the property and the WiFi was down thanks to the rain so we got to spend time without the constant distraction of the online world.


Enjoying the waterfall view

Next day all of us were up early as we had planned to leave immediately post breakfast so that we could reach home before dark. The breakfast was served in the open at a fantastic open air seating and we enjoyed great food with the sounds of nature as background music. Post breakfast we finished our packing and started back to Bangalore relaxed and rejuvenated.

However, the trip had some additional surprises in store for us. After about an hour of driving I realized that my car’s AC Fan had stopped working, and the temperature soon became too hot for me to handle (though Jani loved it). We ended up driving with the windows down, which was fine when on the highway but everytime we had to slow down due to tolls or traffic I was quite miserable. The return journey took us almost 11 hours due to the more frequent stops and traffic.


Breakfast in the forest

By the time we reached home I was exhausted and ended up crashing immediately. We drove ~850 kms round trip and it was worth every minute as the trip was a lot of fun with good food & stay, great company and a fantastic way to welcome 2021!

Wishing you all a Very Happy New Year!

Will write more later.

– Suramya

November 28, 2020

My Backup strategy and how it has evolved over the years

I am a firm believer in backing up my data, some people say that I am paranoid about backing up data and I do not dispute it. All my data is backed up on multiple drives and locations and still I feel that I need additional backup. This is because I read the news and there have been multiple cases where people lost their data because they hadn’t backed it up. Initially I wasn’t that serious about it but when I was in college and working at the helpdesk, a phd student came in crying because her entire PHD thesis was on a Zip Drive and it wasn’t working anymore. She didn’t have a backup and was basically screwed. We tried a bunch of stuff to recover the data but didn’t manage to recover anything. That made me realize that I needed a better backup procedure so started my journey in creating recoverable backups.

My first backup system was a partition on my drive called backup where I created a copy of all my important data (This is back in 2000/2001). Then I realized that if the drive died then I would loose access to the backup partition as well, and I started looking for alternatives. This is around the time when I had bought a CD Writer so all my important data was backed up to CD’s and I was confident that I could recover any lost data. Shortly afterwards I moved to DVD’s for easier storage. However, I didn’t realize till a lot later that CD’s & DVD’s start becoming unreadable quite easily. Thankfully I didn’t loose any data but it was a rude awakening to find that the disks I had expected to keep my data safe were starting to become unreadable within a few years.

I then did a bunch of research online and found that the best medium for storing data long term is still Hard Drives. I didn’t want to store anything online because I want my data to be in my control so any online backup system was out of the question. I added multiple drives to my desktop and started syncing the data from the desktop & laptop to the backup drive using rync. This ensured that the important data was in three locations at any given time: My Desktop, My Laptop and the Backup drive. (Plus a DVD copy that I made of all my data every year)

I continued with this backup strategy for a few years but then realized that I had no way to go back to a previous version of any given document, if I deleted a file or wanted to go back to an older version of a file I only had 24 hours before the changes were synced to the backup drive before it was unrecoverable. There was a case where I ended up having to dig through my DVD backups to find the original version of a file that I had changed. So I did a bit of research and found rdiff-backup. It allows a user to back up one directory to another and generates an incremental backup. So we can recover/restore files based on date range. The best part is that the software is highly efficient, once the initial backup is done it only transmits the changes to the files in subsequent runs. Now that I have been using it I can restore a snapshot of my data going back to 2012 quite easily.

I was quite happy with this setup for a while, but while reading an article on best backup practices I realized that I was still depending only on 1 location for the backup data (the rdiff-data snapshots) and the best practices stated that you should also store it in an external drive or offsite location to prevent viruses/ransomware from deleting backups. So I bought a 5TB external drive and created an encrypted partition on the same to store all my important data. But I was still unhappy because all of this was still stored at my home so if I had a fire or something I would still end up loosing the data even though my external drive was kept in a safe. I still didn’t want to store data online but that was still the best way to ensure I had offsite backup. I initially thought about setting a server at my parents place in Delhi and backup there but that didn’t work out for various reasons. Plus I didn’t want to have to call them and troubleshoot backup issues over the phone.

Around this time I was reading about encrypted partitions and came up with the idea of creating an encrypted container file to store my data and then backup the container file online. I followed the steps I outlined in my post How to encrypt your Hard-drive in Linux and created the encrypted container. Once I finished that I had to upload the container to my webhost since I had unlimited storage space as per my contract. Initially I wasn’t able to because they had restricted my account’s quota but a call to their customer support sorted it out after a bit of argument and explaining what I was doing. The next hurdle I faced was uploading the file to the server because of the ridiculously low upload speed I was getting from Airtel. I had a 40 mbps connection at the time but the upload speed was restricted to 1 mbps because of ‘reasons’. After arguing with their support for a while, I was complaining about it at work and one of the folks suggest I check out ACT Internet. I checked out their plans and was quite impressed with the offerings so I switched over to ACT and was able to upload the container file quickly and painlessly.

Once the container was uploaded, I had to tackle the next problem in the process which was on how to update the files in the container without having to upload the entire container to the host. I experimented with a few solutions and then came up with the following solution:

1. Mount the remote partition as a local mount using sshfs. I mounted the partition locally using the following command: (please replace with the correct hostname and username before using)

/usr/sbin/runuser -l suramya -c "sshfs -o allow_other @hostname.com:. /mnt/offsite/"

2. Once the remote partition was mounted locally, I was able to use the usual commands to mount the encrypted partition to another location using the following command:

/usr/sbin/cryptsetup luksOpen /mnt/offsite/container/Enc_vol1.img enc --key-file /root/UserKey.dat
mount /dev/mapper/enc /mnt/stash/

In an earlier iteration of the code I wasn’t using the keyfile so had to manually enter the password everytime I wanted to backup to the offsite location. This meant that the backup was done randomly as and when I remembered to run the command manually. A few days ago I finally configured it to run automatically after adding the keyfile as a decryption key. (Obviously the keyfile should be protected and not be accessible to others because it allows users to decrypt the data without entering a password.) Now the offsite backup runs once a week while the local backup runs daily and I still backup the Backup partition to the external drive as well manually as and when I remember to do so.

In all I was quite happy with my setup but then I was updating the encrypted container and a network issue made be believe that my remote container had become corrupted (it wasn’t but I thought it was). At the same time I was fooling around with Microsoft One Drive and saw that I had 1TB of storage available over there since I was a Office 365 subscriber. This gave me the idea of backing up the Container to OneDrive as well as my site hosting.

I first tried copying the entire container to the drive and hit a limit because the file was too large. So I thought I would split the file into 5GB parts and then sync them to OneDrive using rclone. After installing rclone. I configured it to connect to OneDrive by issuing the following command and following the onscreen prompts:

rclone config

I then created a folder on OnDrive called container to store the split files and then tried uploading a test file using the command:

rclone copy $file OneDrive:container

Where OneDrive is the name of my provider that I configured in the previous step. This was successful so I just needed to create a script that did the following:

1. Update the Container file with the latest backup
2. Split the Container file into 5GB pieces using the following command:

split --verbose -d -b5GB /mnt/repository/Container/Enc_vol1.img /mnt/repository/Container/Enc_vol_

3. Upload the pieces to Ondrive.

for file in `ls /mnt/repository/Container/Enc_vol_* |sort`; do  echo "$file";  /usr/bin/rclone copy $file OneDrive:container -v &> /tmp/oneDriveSync.log; done

This command uploads the pieces to the drive one at a time and is a bit slow because it maxes out the upload speed to ~2mbps. If you split the uploads and run the command in parallel then you get a lot faster speed. Keep in mind that if you are uploading more than 10 files at a time you will start getting errors about too many open connections and then you have to wait for a few hours before you can upload again. It took a while to upload the chunks but now my files are stored in yet another location and the system is configured to sync to Onedrive once a month.

So, as of now my files are backed up as following:

  • /mnt/Backup: Local Drive. All changes are backed up daily using rdiff-backup
  • /mnt/offsite: Encrypted Container stored online. All changes are backed up weekly using rsync
  • OneDrive: Encrypted Container stored at Microsoft OneDrive. All changes are backed up monthly using rsync
  • External Drive: Encrypted backup stored in an External Hard-drive using rsync. Changes are backed up infrequently manually.
  • Laptop: All Important files are copied over to the laptop using Unison/rsync manually so that I can access my data while traveling

Finally, I am also considering backing up the snapshot data to BlueRay disks but it will take time so haven’t gotten around to it yet.

Since I have this elaborate backup procedure I wasn’t worried much when one of my disks died last week and was able to continue work without issues or worries about loosing data. I still think I can enhance the backups I take but for now I am good. If you are interested in my backup script an extract of the code is listed below:

function check_failure ()
{
	if [ $? == 0 ]; then
		logger "INFO: $1 Succeeded"
	else
		logger "FATAL: Execution of $1 failed"
		wall "FATAL: Execution of $1 failed"
		exit 1
	fi
}

###
# Syncing to internal Backup Drive
###

function local_backup ()
{
	export BACKUP_ROOT=/mnt/Backup/Snapshots
	export PARENT_ROOT=/mnt/repository

	logger "INFO: Starting System Backup"

	rdiff-backup -v 5 /mnt/data/Documents/ $BACKUP_ROOT/Documents/
	check_failure "Backing up Documents"

	rdiff-backup -v 5 /mnt/repository/Documents/Jani/ $BACKUP_ROOT/Jani_Documents/
	check_failure "Backing up Jani Documents"

	rdiff-backup -v 5 $PARENT_ROOT/Programs/ $BACKUP_ROOT/Programs/
	check_failure "Backing up Programs"

	..
	..

	logger "INFO: All Backups Completed Successfully."
}

### 
# Syncing to Off-Site Backup location
###

function offsite_backup
{
	export PARENT_ROOT=/mnt/repository

	# First we mount the remote directory to local
	logger "INFO: Mounting External Drive"
	/usr/sbin/runuser -l suramya -c "sshfs -o allow_other username@remotehost:. /mnt/offsite/"
	check_failure "Mounting External Drive"

	# Open the Encrypted Partition
	logger "INFO: Opening Encrypted Partition. Please provide password."
	/usr/sbin/cryptsetup luksOpen /mnt/offsite/container/Enc_vol1.img enc --key-file /root/keyfile1
	check_failure "Mounting Encrypted Partition Part 1"

	# Mount the device
	logger "INFO: Mounting the drive"
	mount /dev/mapper/enc /mnt/stash/
	check_failure "Mounting Encrypted Partition Part 2"

	logger "INFO: Starting System Backup"
	rsync -avz --delete  /mnt/data/Documents /mnt/stash/
	check_failure "Backing up Documents offsite"
	rsync -avz --delete /mnt/repository/Documents/Jani/ /mnt/stash/Jani_Documents/
	check_failure "Backing up Jani Documents offsite"
	..
	..
	..

	umount /mnt/stash/
	/usr/sbin/cryptsetup luksClose enc
	umount /mnt/offsite/

	logger "INFO: Offsite Backup Completed"
}

This is how I make sure my data is backed up. All of Jani’s data is also backed up to my system using robocopy as she is running Windows and then the data gets backed up by the scripts I explained above as usual. I also have scripts to backup my website/blog/databases but that’s done using a simple script. Let me know if you are interested and I will share them as well.

This is all for now. Let me know if you have any questions about the backup strategy or if you want to make fun of me. 🙂 This is all for now. Will write more later.

– Suramya

« Newer PostsOlder Posts »

Powered by WordPress