October 05, 2018
Last week, I partook in my organization’s Hackathon. The problem statement revolved around improving data quality. It was rather tough, but definitely a rewarding experience because we got 2nd place/~10 teams. This was my 3rd Hackathon, and I was quite surprised and pleased with the results.
Good Points:
- I liked the problem statement. It was definitely relevant, and was a problem where if a group presented a good potential solution, there is some semblance of it being brought to life.
- Environment setup was good. Each time had EC2 instances and an RDS server. My only gripe with this is that configuration is still too tough to get everything working. But that is the nature of our business - one misspelling or configuration error, and the whole thing goes kaboom!
-
Our idea, and slide decks were on point. We pretty much had a top slide deck, and a very useful/practical idea that provided benefit to business. This reminded me of a critical law I learned in school: Akin’s law of Spacecraft Design(https://spacecraft.ssl.umd.edu/akins_laws.html)
20. A bad design with a good presentation is doomed eventually. A good design with a bad presentation is doomed immediately.
Bad Points:
- Presentations were too long. We had 10 minutes for the demo, but with 10 minutes + transition times + teams going over, it was more like 2 hours. Personally, I’m a little desensitized to slide decks. I want to know what the teams have created. What would be nice is to have the team just make a video over the weekend and submit it to the judges. The video would cut out all the transitions, and crystalize and condense the information into a perfect, shiny crystal
- As a team, we spent too much time debating on what to present and what to do. I think most of that time is wasteful - in a Hackathon, there really isn’t any time to build out anything fully fleshed out. I wished that we focused on a minimum, minimum viable product, and moved forward. Nothing fancy(well maybe the front end) going on in the back - just some working prototype to show what the workflow would look like.
Overall, it was a pretty good experience. I had some fun, and I’d definitely do it again. But with some tweaks =/.
September 23, 2018
This weekend I tried to setup a AWS VPN proxy in Tokyo, Japan. My main reason for doing so was I wanted to watch Netflix in Japan. I followed the instructions on this link: https://hackernoon.com/using-a-vpn-server-to-connect-to-your-aws-vpc-for-just-the-cost-of-an-ec2-nano-instance-3c81269c71c2 using OpenVPN.
But there were a couple of modifications I had to make to make this work:
-
You need to uncomment out two lines in /etc/openvpn/server.conf. Every configuration that starts with a ; is effectively a commented out configuration. One for something equivalent to:
push "route 172.31.0.0 255.255.0.0"
push "dhcp-option DNS x.x.x.x" # Push your preferred VPN DNS - google eg. 8.8.8.8
push "block-outside-dns"
(https://forums.openvpn.net/viewtopic.php?t=22039)
- Check if there are any leaks or problems. I’ve checked the websites https://whatismyipaddress.com/proxy-check and https://ipleak.net/ for any traces of my original source ip. There was none.
-
You need a modified command like:
To transfer your generated ovpn file to your local directory
- Use Tunnelblick and the generated ovpn file to connect to your vpn. Make sure to look at the logs for any breaking issues.
So I do all these checks, and I was pumped to try out my new VPN so that I can watch some Netflix in Japan. Like Rick and Morty and Terrace House. So I login to Netflix, and try to start watching a show….
And then, I am greeted by this familiar friend:
This would have been a happy ending if it worked. But it didn’t =(.
How are they finding out that I’m behind a proxy??!
- Apparently AWS has a public JSON of all their ip address ranges (https://ip-ranges.amazonaws.com/ip-ranges.json). The first elastic IP address I put in was on there, but the second one I got was not. Do they have another secret list? There’s really nothing I can do because companies buy blocks of ip addresses with their name on it… No matter what I do, they would still know that the ip is from amazon.com. Unless I go get another obscure cloud service provider to give me an ip…
- Suppose that the elastic ip address that I used is not on their radar. Do they still know that my account was registered in the U.S. and adjust accordingly? It’s not like one person can connect from the U.S. at 1 PM and suddenly connect from Japan at 5 PM. Unless they can teleport…
Anyhoo, I’m a little disappointed that I can’t go on Netflix Japan. If they are using some techniques that align with #2, then a VPN is virtually useless. It leads me to wonder if there are other sophisticated techniques to detect proxies. I wonder, if at the packet level, if there is a difference between a packet sent from the U.S. vs. a packet sent from Tokyo. Even just by timing, wouldn’t you be able to figure this out?
For example, this is a simple ping to Google. This is with vpn turned on and off:
#VPN ON
PING google.com (172.217.161.78): 56 data bytes
64 bytes from 172.217.161.78: icmp_seq=0 ttl=43 time=187.722 ms
64 bytes from 172.217.161.78: icmp_seq=1 ttl=43 time=186.720 ms
64 bytes from 172.217.161.78: icmp_seq=2 ttl=43 time=189.030 ms
64 bytes from 172.217.161.78: icmp_seq=3 ttl=43 time=186.826 ms
64 bytes from 172.217.161.78: icmp_seq=4 ttl=43 time=188.989 ms
64 bytes from 172.217.161.78: icmp_seq=5 ttl=43 time=186.898 ms
64 bytes from 172.217.161.78: icmp_seq=6 ttl=43 time=189.009 ms
64 bytes from 172.217.161.78: icmp_seq=7 ttl=43 time=199.305 ms
64 bytes from 172.217.161.78: icmp_seq=8 ttl=43 time=192.338 ms
#VPN OFF
--- google.com ping statistics ---
9 packets transmitted, 9 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 186.720/189.649/199.305/3.794 ms
PING google.com (172.217.161.46): 56 data bytes
64 bytes from 172.217.161.46: icmp_seq=0 ttl=48 time=164.479 ms
64 bytes from 172.217.161.46: icmp_seq=1 ttl=48 time=163.677 ms
64 bytes from 172.217.161.46: icmp_seq=2 ttl=48 time=166.483 ms
64 bytes from 172.217.161.46: icmp_seq=3 ttl=48 time=163.280 ms
64 bytes from 172.217.161.46: icmp_seq=4 ttl=48 time=163.296 ms
64 bytes from 172.217.161.46: icmp_seq=5 ttl=48 time=164.817 ms
64 bytes from 172.217.161.46: icmp_seq=6 ttl=48 time=162.720 ms
^C
--- google.com ping statistics ---
7 packets transmitted, 7 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 162.720/164.107/166.483/1.179 ms
(I know the ip is different, but they’re location check out to California).
If Netflix measures the round trip times of packets, calculate things like standard deviation, percentiles of packets, total response time between actions, etc and feeds it into some statistical model, they can easily figure out that there’s a high chance that this person is behind a proxy.
As for this VPN, unless I have a breakthrough, I guess I’ll be sticking to Netflix US for the time being. At the very least, I now have my personal VPN in Japan. The silver lining to all this is that my personal VPN is wicked fast =D.
August 24, 2018
I came back home from work one day this week, and I really didn’t want to code anymore. So I didn’t.
So I lounged around and watched Netflix.
But then this thought occurred to me:
Is “I don’t feel like doing it” even a valid excuse?
And what is the reasonable justification for not doing what you should be doing?
I don’t feel like working out.
I don’t feel like doing homework.
I don’t feel like taking care of my kids.
Shouldn’t be:
I can’t work out because my shins are busted?
I can’t do my homework because I’m tired.
I can’t take care of my kids right now because I’m sick.
In summary, I don’t think emotions should dictate our actions. There’s going to be days when you’re going to be super motivated to go for that run, and code the night away. At the same time, there’s going to be crappy days when you really don’t want to take another step after a lousy day at work, nor make another damn stroke on the keyboard. Not because you’re tired, but because you just don’t feel like it.
Why should you work towards your goal only during your good days, and halt during your bad days, solely based on your emotions?
I think you just have to keep at it. Be disciplined. Take just one more stroke. One look. One more step towards your goal.
There are times when the weather is perfect, and the sun is right up in your face and day is gorgeous. But aren’t there days that are so murky that you start to question the existence of a light source? Regardless, the sun is still there. It still rises from the west and sets in the east.
Similarly, you just have to work on yourself. Chip away the sculpture you are making one piece at a time. Because the moment you drop the chisel, you gotta pick it up again. It’s so much better to hold onto it, chip even a minuscule piece of what you want to accomplish rather than let your muscles relax and lose all tension and focus.
August 06, 2018
What is the most depressing thing about maintaining friendships is when you realize that the other party never held your relationship to the same value that you’ve held it. Or that the friendship never had any substance except for the fact that it was two people shooting the shit.
The evidence of how your friend who you thought you were close with breaks down when you’re in a time of trouble. Or when you’re in a position of weakness and low status. The people who you’ve thought would help you, don’t, and let you down. Maybe they’ll look down upon you.
There is a custom in Korea where if your mother or father dies, your friend visits you in the funeral home, pays respect with a bow, and gives you some money to help offset the expensive cost of running a funeral.
My dad and I traveled for a funeral in Korea for my grandmother, and since we live in the States, the funeral shed light on who were my father’s closest friends. This is because my dad doesn’t live in Korea anymore(although he keeps in touch with his friends), and there’s hardly any chance for him to come back and go to the funeral of all his friends. That means that there is probably NO WAY for him to visit any of his friends’ parents funerals, and pay back the money he received.
My dad said he was disappointed that some friends whom he held in high regard never came to pay respects for his mother, because they know that it’s unlikely that they won’t be able to recover the money they give him. That it was an asymmetric relationship, where you’d go to lengths for your friend, but he wouldn’t do the same for you. Others drove 3-4 hours to pay respects to my grandmother.
But now he knows.
There was a situation for me as well when I was in a pretty rough spot, and I needed a place to stay for one night and be picked up from a train station. Only one friend came through for me. And I’m deeply thankful to him for that.
I feel that I’m encountering this situation once more, and this is causing me to re-evaluate my old relationships, and makes me want to think about what friendships really are. Are they of convenience? Are they of support and encouragement? Is it, some degree my fault? And how do you even become close friends in the first place?
To me, it doesn’t matter how many people I know - I’d rather have few close friends who are like hail that pounds on the sidewalks than a bunch of snowflakes that melt as soon as they touch the ground. But I don’t want to be hail for those who won’t be hail for me.
July 15, 2018
I have a desktop computer I’ve built back when I was in college, and it’s been nothing but awesome. A couple of months back I used it so that I can log in through ssh using a couple of neat programs. I set this up, and I thought it was the coolest thing ever.
Suppose you’re in Thailand, and you want to be able to login to your GPU’d desktop. You can! But you just have to make sure you login using Linux… Yesterday, I tried to login from a local library to play around with some machine learning. There’s a dataset I’m looking at, and it’s >10 GB! There’s no way my puny laptop can handle that much processing. It’ll heat up.
Anyhoo, I tried it again after I haven’t used the GPUs in a while, since I’ve been up to making an Electron app, among other things. I tried to do it again and I was just so last.
After hours of fumbling around, and reading a bit more on the actual topic, I was able to get it working again. Last time, I wanted to write a post for myself and posterity, but never got around to it. But I’m going to do it this time!
Approach
First, I’ll explore the theory behind each broad strokes procedures in diagram form(as pictures are worth a thousand words) then I’m going to do the entire procedure on my laptop with another computer being the server, and put screenshots and commands in step by step.
Part One: Setting up No-IP
Part Two: SSH
Part A: Session Key Establishment
We don’t do any work on this, the ssh program does all of this for us. But I decided to read a little on it anyways and made this diagram, which will be revised if I get around to recalling all the actual details from my undergraduate days…
.png)
Part B: Authentication
To make sure that’ we know we’re making contact with the appropriate party, after the session key is established, you can login with userid and password. In lieu of the password, you can generate what’s called a public-private key pair.What’s nice about this is that you don’t have to type those dank passwords and get error messages and crap. You just log in.
Public Private Keypair
To do this, you generate a public private key pair. The rule is: you keep private key yourself, but are free to share the public key with anyone.
You can encrypt a message with a public key, and only decrypt with a private key. Example:
.png)
SSH Authentication
Now this part is the most relevant for our setup. I’ll try to number the steps, and afterwards, go on my computers and document the steps required.png)
Application
Finally, this is where the real fun starts.
- Because the IP provider doesn’t give us a permanent address, it could change at anytime. We need an intermediary to send our ip address, and this intermediary needs to reroute the IP address back to the requester. It kind of looks like this:[Picture to be added later]
-
To do this, we need a service to constantly update the IP that our internet service provider feeds us. For this, we can use a service called No-IP(https://www.noip.com/). Make an account, and create a hostname:
Alternatively, you can get a Hostname and domain if your router provides you one. for my router, Netgear allows me to have a separate Dynamic DNS Service name, which is described in the section below.
-
You also need to setup your router configurations:
A. For this, I went to my routerlogin.net -> went to advanced-> advanced setup, and opened up port 22 for ssh. TCP/IP ports(if I recall correctly, for this I went to my router’s webpage routerlogin.net, went to advanced and filled out the necessary details)
B. Optionally, you can setup the hostname under the router company’s sub-domain name instead of using one from no-ip (which has restrictions on the number of DNS routings, and unintuitive domain names).
-
Download and install the software on the server for no-ip. This is used to ping the no-ip DNS servers and update the entry to point to the local ip address. Follow the instruction here: https://www.noip.com/support/knowledgebase/installing-the-linux-dynamic-update-client-on-ubuntu/
Below are the commands to install and update the client:
-
Install ssh authentication for your server. That way, you’ll be able to login without specifying the password. First, you want to generate a public and private key pair on the client machine:
ssh-keygen #generate client key
cd ~/.ssh
-
Next, if you’re on a mac, add the private key to your keychain. I’d imagine you’d have to do some command to add the private key to your ssh-client, depending on your operating system.
You must do this EVERY time you restart the machine, since you have to load the key into memory. It may be good to just put it in your .bashrc file so you don’t have to remember to do so.
- You have to then transfer the public key into your server. This can be done using a flash drive, a secure storage service, or login directly to the user with password, and then transferring the public key: I think this instruction is a good one(https://www.linode.com/docs/security/authentication/use-public-key-authentication-with-ssh/), but I used a flash drive to transfer battlestation.pub into my server computer.
-
Next, add the public key to your ssh server.
cat battlestation.pub >> .ssh/authorized_keys
Finally, start the ssh server. On your server(additional instructions: http://ubuntuhandbook.org/index.php/2016/04/enable-ssh-ubuntu-16-04-lts/
sudo /etc/init.d/ssh start
sudo service ssh restart
-
Modify your ~/.ssh config to include your new domain. For example: mynewdomain.mynetgear.com
Host bs
UseKeychain yes
AddKeysToAgent yes
Hostname yoursubdomainname.mynetgear.com
User me
IdentityFile ~/.ssh/battlestation
-
also remember to clean our your known_hosts file in .ssh, if you’ve attempted this more than once and have unclean known__hosts file:
ssh-keygen -R hostname
-
Finally, I was able to login! Now I’m able to login to my desktop to do machine learning on anywhere in the world!
References