Hacktivate

Noise blocking earmuffs...check
Music...check
Children locked away in kiddie jail...check

Using autossh and reverse ssh tunneling to bust out of NAT'd networks and firewalls automatically.

In another lifetime I blogged about how to reverse an ssh connection. This was by far my most popular post ever. It earned enough in google ad money to cover my hosting costs. Those were the days. Those days are gone, and now lots of people have blogged about using reverse ssh tunnels. It's time to take it to the next level and make reverse ssh tunnels easy and ubiquitous.

I want all of my machines to be setup with an ssh tunnel whenever one of them finds an internet connection. That way I can always access all of my machines no matter where I, or they are the world. I want this even if they are in some high security firewalled corporate prison or if they are sharing the same IP address as the other million NAT'd users tethering internet through their phones. This should happen automatically and without fuss.

To do this you need to have ssh access to a machine that can be seen from the internet, and the ssh daemon machine needs to have the GatewayPorts option set to yes (this is not the default). See my old post for more information.

Here's the recipe.

(Update! I have made this much easier by scripting the whole process, see the bottom of the post!)

Create a file:

/etc/network/if-up.d/reverse_ssh_tunnel

(you are using Ubuntu, right?)

Put this in it:

#!/bin/sh
# ------------------------------
# autossh reverse tunnel on boot
# ------------------------------
# See autossh and google for reverse ssh tunnels to see how this works

# When this script runs it will allow you to ssh into this machine even if it is behind a firewall or has a NAT'd IP address.
# From any ssh capable machine you just type ssh -p $PORT_MIDDLEMAN_WILL_LISTEN_ON localusername@middleman

# This is the username on your local server who has public key authentication setup at the middleman
USER_TO_SSH_IN_AS=change_me

# This is the username and hostname/IP address for the middleman (internet accessible server)
MIDDLEMAN_SERVER_AND_USERNAME=luser@example.com

# The following two numbers can be whatever you want, but need to be unique if you have multiple reverse ssh tunnels
# Port that the middleman will listen on (use this value as the -p argument when sshing)
PORT_MIDDLEMAN_WILL_LISTEN_ON=11829

# Connection monitoring port, don't need to know this one
AUTOSSH_PORT=27554

# Ensures that autossh keeps trying to connect
AUTOSSH_GATETIME=0

export AUTOSSH_PORT AUTOSSH_GATETIME

su -c "autossh -f -N -R *:${PORT_MIDDLEMAN_WILL_LISTEN_ON}:localhost:22 ${MIDDLEMAN_SERVER_AND_USERNAME} -oLogLevel=error -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no" $USER_TO_SSH_IN_AS


Make sure that you fill in the values to match your own logins (send me a comment if you are confused about what to put in).

Now whenever the machine acquires an internet connection it will run the above script (that is why we put it in /etc/network/if-up.d). Then from any internet connection you can ssh -p 11829 USER_TO_SSH_IN_AS@MIDDLEMAN and you will get forwarded to your own machine.

You should put this on all of your machines. That is all.

Oh one more thing. For this to work you need to have passwordless public key authentication working between your machine and the middleman. Use this:


ssh-copy-id


It will take care of all of the gory details of copying and concatenating your keys so that you can ssh in without typing anything.

This blog post is a mess and needs some serious revision, but I needed to post it, if for no other reason so that I could find the info when I needed it!

Update

I put everything you need to set this up into a script on github. If you have a server out on the internet with the GatewayPorts option turned on, then all you need to do is the following:


wget http://github.com/mikeymckay/reverse_ssh_tunnel/raw/master/setup_reverse_tunnel.sh
chmod +x ./setup_reverse_tunnel.sh
sudo ./setup_reverse_tunnel.sh
:

Someone is doing something nasty. That's the title of my favorite Linux error message. It's to stop another computer from emulating another computer in order to try and steal passwords. They call this a man in the middle attack.

Unfortunately, I am always caught doing something nasty. It's because I work with a lot of Linux machines, and I setup VirtualBox or VMWare servers, and each new server looks like an untrusted machine when I try and ssh to it. This always results in me getting this:


@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
d7:63:c7:d5:65:55:33:fc:bf:b5:d6:f9:c1:a5:2e:d6.
Please contact your system administrator.
Add correct host key in /home/crazy/.ssh/known_hosts to get rid of this message.
Offending key in /home/crazy/.ssh/known_hosts:66
RSA host key for [localhost]:2222 has changed and you have requested strict checking.
Host key verification failed.


So then I have to open up the known_hosts file, find the line, delete it and then reconnect.

After years of doing the nasty like this, I finally figured out how to use the ssh config file to solve this problem (and do other cool stuff too):

From ~/.ssh/config:

Host chits
Port 2222
HostName localhost
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
LogLevel error
User chits


This combines a couple of nice things. Here is the stuff that protects you from the nasty:


UserKnownHostsFile /dev/null
StrictHostKeyChecking no
LogLevel error


It looks at the null file to check for a match on the new server's key. It eases off the strict-o-ness level, and it doesn't bore with warnings that you already know.

The other stuff just makes it require less typing to login to my virtual servers which listen on strange ports and require funny user names.

Sinatra is an awesome tool for creating dynamic websites really really fast. But their deployment recommendations just don't match their overall KISS ethos. So here is my recommendation if you want to deploy a sinatra app really quickly:

If your sinatra app is in:

/var/www/sinatra/cool_app.rb
Just insert in /etc/rc.local (before the "exit 0" line)

nohup /usr/bin/ruby /var/www/sinatra/cool_app.rb&
And then run the same command. Now your application is running, and the next time your system reboots it will start up again.

---

Update. I probably wouldn't recommend doing this anymore. The world has come a long way since then. It's still a pain to do this right (using something like apache passenger), but you can do a lot better than the above by doing a few more things:

Install shotgun and thin:

sudo gem install shotgun thin

And then put this at the end of /etc/rc.local:

/usr/local/bin/shotgun /var/www/sinatra/cool_app.rb&


It should work the same as before, but also be able to handle more than one request at a time.

A friend recently challenged me with the following hypothesis: Open source is something that is valuable to programmers but when programmers are writing open source software they have no incentive to build software that is useful for end users. Because of this, we we end up with overly complex software that may not even address the need of the end users and hence the original intent of the project.

After thinking about this for awhile I have decided that I agree that there is a serious disconnect between software developers and end users, but I don't think open source has much to do with it. Programmers are good at logical thinking (without emotion), understanding complex documentation (pages of text), and holding multiple concepts and equations in their heads at the same time. In fact they tend to thrive on this kind of stuff. Case in point, I was just looking at a product that allows you to remotely control someone else's PC. Let's have a look at two images on their site:


Whether you are a computer programmer or not, that is a terrible attempt to communicate something. The only thing it communicates to me is that this setup is complex. Perhaps that is how the programmer's mind visualizes what he is building, and that is fine, but let's keep our private parts (brain dumps included) to ourselves, eh?



I thought this second image (they are both on the same page) was pretty funny. Instead of explaining that the software can help your mom when she "loses the internet" by letting you control her computer from afar, the developer decides to show you how you can recursively connect to yourself and create an infinite number of desktops within desktops. "Whoa dude, isn't it crazy what I can do?? I feel like my brain is going to explode!" Sorry, elite hacker dude, it's not cool. It's lame and I am sorry that normal people will never use your software because most people don't like software that makes their brain explode, they like things that help mom. Most people, but not computer programmers. The more head-xplode-dacious the better for geeks!

This problem gets compounded when you have programmers in DC "solving" problems in Africa or the Philippines. Not only do we have the programmer/normal person gap, but we also have a cultural gap (take it from me, never use the word stupid in the Philippines) and a geographical gap (body language and timezones matter). The greater the gap, then the more useless the stuff we end up with is. The more time programmers spend in country the more we can reduce this gap. Or even better, find some programmers from the country itself. This can improve things a lot, but you still have the classic gap between programmers and end users.

The way to solve this is to have programmers never design anything that a user will see, ever. Instead you have designers design stuff and programmers implement it. Back in the dot com days I worked for a big technology consulting firm and every project had at least one business strategy person, a designer and a computer programmer. We created some world changing stuff, but we were expensive, spent a lot of money on flashy artsy offices and flamed out. But I worked with designers and learned to understand their value (and my own shortcomings in that area).

Macs and iphones aren't programmed in California - they are Designed (check the back of your iphone). I hear that Apple has heavily relied on their Silicon neighbor IDEO to do a lot of their design work, or more likely they probably just hired a bunch of IDEO people. IDEO is a pure design firm, they spend most of their time watching and talking with users, and the other part of their time is spent trying to think outside the box by crossing and combining design patterns from disparate genres and seeing if they can come up with something useful. I just installed an iphone app called Party Whistle designed by the IDEO guys. It is the antithesis of the above examples:



When you launch the app you see a party whistle on the screen, and a little icon recommending that you blow into one end of your phone. When you do his, the whistle expands and makes funny noises and all sorts of wacky stuff. Emphasis on wacky. Nothing useful. It doesn't help plan a party or coordinate party favors, it is just wackyness designed elegantly and simply. This is what you get without programmers.

What we really need is to take the brilliant visual ideas of designers and connect them with innovative plumbing that programmers do.

Usually we do this backwards, and we give the problem to the programmer and they plumb it, then they make it so that they can use it, then they try and make it look nice. With a single designer or artist or business plan you have nothing. But with a single programmer you can usually create something, even if that something is just a shadow of what it could be. It's this shadow world of foggy potential that the programmers tend to fester.

How does this relate to open source? It doesn't really. The best programmers hate to create boring stuff like menus and reminder windows because all of that stuff has been done before. They would rather reuse someone else's code so they could focus on doing the really challenging (brain exploding?) part. This is how programmers are wired, so if they aren't allowed to use open source they will be less happy and will be forced to waste their time on reinventing the wheel. Whether it is open or not will have no impact on usability.

Open source lets programmers stand on the shoulders of giants and build the stuff that they think is important and cool. As a result open source has little to do with cost (of any kind) and more to do with creating an easy way for programmers to share, communicate and collaborate. I prefer working with open source software because I can find other developers trying solve the same problems on IRC. I can google for documentation and wiki my own thoughts to it so that others benefit and see how cool I am. I can join mailing lists. It also means that open source programmers write their code so that others can read it and use it and modify it easily. All of this leads to high-productivity, innovation and personal fulfillment for open sourcers.

It doesn't lead to good user centric design.

Yet open source programmers are figuring out good ways to work together. Open source fosters that, and that is why the innovation is coming out of open source. Open source enables programmers to work on the innovative stuff. Tools like cucumber (story based testing) and github (facebook for geeks) and others are further helping programmers spend their time on innovation, and less on the debugging and already solved problems.

The challenge then is to figure out how to use the open source approach so that it can bridge the geeks with the artists. Imagine if designers could easily express their prototypes without having to waste a lot of time explaining it to programmers. When artists can hook into code as easily as they can doodle a design, then we will have a renaissance that spreads beyond just the geek elite.

[caption id="attachment_285" align="alignleft" width="324" caption="William Kamkwamba on Good Morning America"][/caption]

All things are possible when your dreams are powered from the heart. -William Kamkwamba

William Kamkwamba, whom I first blogged about three years ago continues his amazing ascent. Where there was nothing but a drought laden field he built a tower of scrap wood, climbed it and built a windmill out of flatten plastic pipe and wire he wound himself. He climbed the stage at TED and has inspired thousands. As an adult, he went back to school and climbed the learning curve of academia despite a rudimentary village education. This morning he was beamed around the world as he was interviewed on Good Morning America. His book, The Boy Who Harnessed the Wind comes out tomorrow, when he will begin to climb the best sellers lists. I can't wait to read it. I am even part of it! I am mentioned five times according to Amazon's Search Inside. How cool is this:

One of Soyapi's bosses, a tall American named Mike McKay, liked the article about my windmill so much that he wrote about me on his blog, Hacktivate.

Thanks for mentioning me William - and congratulations!!!

Challenges

Bringing technology into health clinics can benefit healthcare workers, patients and public health workers with improved decision making support and better quality data. While there are many benefits offered by a successful system, there are many challenges in creating and sustaining a successful implementation. This document will discuss the problems of electricity, viruses, repurposing of equipment, theft and will also discuss solutions for these problems.

Electricity

Even in developed countries electricity is unreliable. Equipment wears out, the infrastructure is overburdened, lightning strikes, people forget to pay their electricity bill. In developing countries like the Philippines, particularly in rural environments, the problems are amplified. Instead of power outages occurring just a few times a year and lasting 5-10 minutes, developing countries experience monthly or weekly power cuts lasting for hours at a time. The approaches used by developed countries for handling their relatively short, infrequent power cuts work are optimized for their situation, and are hence a poor fit for developing countries. UPSs are the standard solution for desktop computers in developing countries. Their components are designed for infrequent use and are optimized for delivering 15-30 minutes worth of electricity at the power level a desktop computer requires. When these UPSs are deployed in developing countries they are unable to sustain equipment through the multi-hour power cuts and they tend to wear out very quickly due to the high frequency of use that they receive. Developing countries require a reliable power solution that is better aligned with their realities.

Viruses

Viruses are a major problem for Microsoft Windows users, which are the majority of computer users. While recent Microsoft products have improved their security models, all users of Windows are strongly recommended to use antivirus software. But for antivirus software to be effective it must be constantly updated in order to avoid being infected by the new viruses and attacks that appear daily. Because virus updates usually cost money, and require reliable internet connections, updates are often forgotten. This problem is particularly serious in developing countries. Even if money is available for updates, a reliable internet connection is often not available, so antivirus systems are often out of date.

Some virus infection techniques are also particularly effective in developing countries. Because few people have their own computer and even fewer have access to the internet, USB flash disks are used heavily. People carry their personal data on their flash disks, and then plug them into computers that are shared - either at work, an internet cafe or at friends' houses. As these disks move from computer to computer they become infected and end up attacking every computer that they are used on (which often have no or an old antivirus program). This combination of out of date virus software and heavy use of USB flash disks means that computer viruses are rampant in developing countries. Pirated movies and software which are also widely used and distributed in developing countries are another common virus spreading medium.

Repurposing

Computers are multi-functional devices. They can be used for writing reports, storing patient information, watching movies, looking up useful information and playing games. While it is great that a computer can do so much, it often means that a computer purchased for a specific project rarely stays dedicated to this project. In some cases, repurposing or multi-purposing a machine can improve the overall goals of a project. But when a machine that performs a critical function isn't available because it is being used for another purpose or is not functional due to a problem caused by another program that was installed then the project can be compromised. For a health clinic, if a computer is used for patient care and real-time recording of patient information then it is critical that the computer is always available and working reliably.

Theft

Finally, computers are easy targets for theft. They provide a lot of value in a small package. Whether an inside job or a simple break-in, a personal home or a workplace, the computers are often the first items to be stolen. Computers in rural, poverty stricken areas make for tempting targets.

Solutions

These four problems: electricity, viruses, repurposing and theft, must be carefully considered for any kind of computer based system to be successful in a developing country. Now that the problems have been described, this document is going to explore solutions to these four problems.

Electricity

Frequent multi hour power cuts require something other than UPSs. Developing countries often turn to diesel generators, which are poor choices because they require special maintenance and fuel. I saw a generator grind to a halt because no one changed the oil and I know that given the choice to keep fuel in the generator or to go rescue a sick patient with the ambulance, that the ambulance always wins. Even with a generator there will be a few second gap without electricity that will reboot all of the machines. How can we move beyond the generator paradigm?

The first and most important approach is to reduce the amount of electricity that is required for a computer to operate. Desktop computers tend to consume about 150-200 watts of electricity while they are being used. Laptop computers will consume only 10-30 watts of electricity. This means that for a laptop computer to operate it needs just a a fraction of the electricity that a desktop computer needs. Clearly laptops are a good choice when electricity is a scarce resource. The other reason that makes laptops a good choice for developing countries is that they come with an optimized battery back up system already built-in. Laptops are designed to handle frequent, multi-hour power interruptions and are hence a very good fit for the kind of situations found in developing countries. The combination of low power requirements and battery optimization means that modern laptops will run for 4-10 hours on battery power. Laptop cost has also dramatically fallen making them cost close to the same as a desktop computer. For roughly the same cost as a desktop and UPS a laptop can be purchased that will last more than 10 times longer. Electricity problem solved.

Viruses

The virus problem can be mitigated by using a non-Windows operating system and or eliminating the paths of typical infection. Ubuntu Linux is a good alternative to Windows because it is based on a very mature and proven security model that stops many of the viruses that affect Windows systems. Specifically, most activities take place within a secured sandbox that requires a password to be bypassed. Hence, viruses have a very difficult time infecting any core system processes.

Open source software is less prone to viruses than the closed source alternatives. Closed source software is distributed without access to the underlying source code (hence "closed source"). Without access to the underlying source code it is impossible for the person using it to know how the software works. This hidden layer of execution is where viruses operate. Open source software reveals everything, there is no hidden layer for viruses. While having the ability to audit software doesn't guarantee its reliability, the collaborative approach used to develop it means that malicious code rarely finds its way into open source software. Ubuntu Linux is an open source operating system and all of the additional software that most people use (like web browsers, email clients and word processors) is also open source. Practically, this means that Ubuntu Linux users have very little chance of being infected by a virus.

The other approach that can be useful is to disable flash disks, as these are the primary infection path for computers in developing countries. Of course being able to use USB disks to transfer information is an important feature, especially in developing countries, so the benefits must be weighed against the costs. There are currently no known viruses that infect Ubuntu Linux via USB disks.

Repurposing

For mission critical computer operations like the delivery of patient care, computers should be made single purpose. They should be configured so that the only function that they can perform is the function that they were purchased for. This "appliance model of computing" follows the same strategy behind household appliances like a toaster, which are dedicated to reliably performing a single task. When an appliance computer is turned on, it should immediately launch the software required for the application. The software should take up the entire screen, and the user should not be able to close the application or access a menu to launch a different application. This sort of approach will ensure that the computer is always available for the task it was purchased for. Because other programs can not be accessed or installed it means that there is no chance of the system becoming corrupted and usable because of some problem caused by another problem. This also greatly reduces the likelihood of virus infection.

Theft

Theft can be avoided by making the computers harder to physically steal, and also less desirable to steal. Physical security like strong doors/locks is obvious. Using cable locks is another important approach, although these can easily be cut by dedicated thieves. Alarmed cable locks are even better. Physically bolting machines (laptops/desktops) to desks is another good and simple strategy if you can find a good way to attach the bolt to the machine. Making a machine less desirable for theft is also a good approach. An appliance computer that is unable to play games, access the internet or a word processor is going to be less desirable for a thief who knows about the machine. Appliance computers can even be configured to operate without a hard drive, so that they will only turn on when they are in the clinic. This approach is called "net booting" and is a good strategy for reducing desirability (and also reducing system complexity as all systems will use the exact same software).