PyTorch Issues with PopOS

I’ve been using PyTorch, and there are a couple of things that have gotten me. I’m writing this down for future reference.

It’s highly recommended that you do NOT screw with other linux distributions other than Ubuntu. It will make a lot of things miserable. For example, I switched to PopOS which I think is built on top of Ubuntu, but still had issues. I couldn’t find good resources to solve the issues.

For PopOS, they seemed only have CUDA versions up to 11.2. But As of April 2022, We’re up to CUDA 11.6

I think this is the appropriate steps to remove PopOS dependencies, and install Ubuntu drivers.

#remove prior Pop-Os nvidia stuff
sudo apt remove nvidia-cuda-toolkit
sudo apt remove system76-cudnn-11.2
sudo apt remove system76-cuda-11.2
sudo apt remove system76-driver

sudo apt install nvidia-driver-510
sudo apt install nvidia-cuda-toolkit

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

One super critical thing that went past me, is that you cannot simply swap out the driver versions like this, and change from 11.3 to 11.2[2]

conda install pytorch torchvision torchaudio cudatoolkit=11.2 -c pytorch 

This will download the latest pytorch version, and it may be not compatible with cudatoolkit-11.2 You have to downgrade the PyTorch version.

[1] https://askubuntu.com/questions/1288672/how-do-you-install-cuda-11-on-ubuntu-20-10-and-verify-the-installation

[2] https://discuss.pytorch.org/t/torch-cuda-is-not-available/74845/27

Principles for Everyday Problems

When you’re looking to solve an everyday problem, think through things just a little bit harder. I’ve been thinking about a couple of principles that could help with everyday problem solving. Of course, all this is retrospective, so you have some kind of hindsight bias built in. But the principles are: surveying a couple steps ahead, subsampling, decision tree logic/sequencing your actions, and expected value calculations, or cost to perform an action.

A couple of problems have risen in my life. A little bit of thinking in advance could have saved a ton of time and resources.

Case 1: Cold Pho

When you order pho, you get some raw meat. You put the raw meat into the bowl, and the hot soup cooks the raw meat But when you order take out, you get the soup in a to-go plastic container which does not guarantee hotness.

After you excitedly bring the take-out home, the natural course of action is - you take the noodles, vegetables, the meat, and pour the soup. If the soup doesn’t cook the raw meat, then you put it back into a pot, heat up the entire thing to make this process work.

This is dumb. This is what you could do instead. Take a spoon and taste one spoonful of the soup. If it’s hot, stop. If you’re still not sure, first, you put the 1 slice of meat in the bowl. Then you pour small amount of soup, and see if it cooks the meat. If it is good, then you can dump the entire thing.

The natural disadvantage of this is that temperature decays extremely fast. So while you are sampling, you could lose the temperature of the entire soup.

Case 2: Leak in an Apartment

This happened to a friend recently. cough

There is a leak in an apartment complex on the first floor. The plumber is summoned! The plumber decides to cut up the wall in the 1st level apartment, check for leaks. Nope. Goes to the 2nd level apartment, cut open the wall, then check for leaks. Nope, not leaking. Now he goes to the 3rd level apartment, and checks for leaks. Yup, it’s there. Thankfully he doesn’t have to open up the wall.

This is stupid if you think about it. You’re cutting up 2 walls before you diagnose the solution. Why didn’t he sample the problem range before taking action?

This is how it could have gone.

You ask the 2nd floor and the 3rd floor residents, “Hey, is there anything that happened in your bathroom recently that could indicate a leak? Can I come check?”

If they’re not home, you could call management: “Hey do you have number for unit 20X, 30X?”

You call them, and say, “Has there any leak of any kind, or change recently”? Looking breadth-first in this case is an optimal approach.

Maybe the plumber could have easily diagnosed the solution there, and finished up the job. But hey, maybe his incentives are different. Perhaps he wants to optimize for increasing his hourly rate! But if he is not mischievous or unethical, this is the wrong approach.

Anyways, if the plumber asked the resident on the 3rd floor maybe he could have spotted the issue a lot faster without wasting so much time.

The other thing is strategy and the sequence in your provide the options, as well as calculating the discount factor.

So the plumber finds the problem on the 3rd floor. He says he could fix it, gives him an hourly rate and everything. The thinks about it, has no information on whether or not it’s expensive or not, has no information on the rates of plumbers(thinks it’s high), and cannot do any research since he has to make the call now.

As the friend doesn’t have doesn’t have enough information, the friend declines the repair.

Now at this point, the plumber could say, “Hey I’m already here. Instead of charging you a rate $X, I’ll do it for $0.9X. “, or $X - cost it takes to get to another location.

I’m almost certain that if you mention the magic word, “discount”, the plumber would’ve had a much higher chance of converting my friend to take the plumber’s labor. Hell, he could have given a smaller discount and it still could have worked.

Notice how this is different from if the plumber gave a discount first. It’s better for him to save the discount for a rejection, since it will increase his expected value(if he does this move multiple times, there’s that assumption built in there…)

But the plumber didn’t make this move…

Case 3: Production Bug

At work, we had a bug in production. So 5 - 8 of us are on a call, we proceed to try to find why an api request is getting double called. An hour in, we find the most probable cause - a UI issue, but we are not certain. After we found the most probable cause, we proceed to spend another hour+ trying to diagnose all other possible problems.

This is where I’m dissatisfied with how it went. If the possibility of the bug we found being the root cause is 80%, and the prod bug isn’t mission critical, then isn’t it better to assign someone to fix it, and then come back to it? Why would you still be exploring for the other 20%? It makes no sense.

You fix the bug with the 80% probability first, then you explore the remaining options. Because finding the 20% is much, much harder. Just all marginal utility. Another advantage you get from this, is this. Ff the bug takes a day to fix, then developers have 1 night to sleep over it to let the brain parallel process.

To sum it up mathematically:

80% probability of bug is the issue * time it takes to fix it + time it takes to look for the other 20% + 20% probability * cost it takes to fix the issue.

Could have rearranged this to consume only the 80% probability of bug is the issue * time it takes to fix it….

In this case, it’s a bit opposite of the case of the apartment leak, except the cost to look for the other 20% is HUGE, while the cost to sample a solution at a higher overview before diving in is extremely low.

Summary

Maybe it’s the computer science in me, but for every action or a problem, there is some theoretical magical, optimal moves that will get you to the right solution. This is the Move of God. The steps that God would take are there in the solution space, except we probably don’t know… Instead of taking the first action that comes to mind, maybe we should explore more and think through alternative approaches.

Case for Crypto

My experience with cryptocurrency so far.

2012

Back when I was in college, my childhood friend and I were walking down one of the main walkways from our school. We had gotten in touch after a long time. While he was walking, he asked me,

“Have you heard about Bitcoin?”

“I’ve vaguely heard about it. Some kind of internet money… ?”

“I put some of my tuition money in Bitcoin. But man, it goes up and down so much. What do you think about it?”

“I don’t know.”

And I never bothered looking it up.

Later, we lost touch and I have never been able to get back in touch with him. Part of me thinks he has now sailed off into the sunset with his bitcoin earnings, and drinking a pina colada while baking himself in the Caribbean.

2017

In 2017, I also met a friend at a coding workshop. I asked him what he’s been up to.

“Dating and Cryptocurrency dude.”

This was when Bitcoin was at maybe $1,000-$2,000 dollars. My memory from our conversation is that he also held a significant holding. We both stared at the computer together and looked at some cryptocurrencies.

“Ripple? Yeah that one might be good. But I’m not sure if will get adopted.”

Curious, I went back home, read Satoshi’s white paper. I wasn’t blown away by the paper or anything. The computer scientist in me thought, “Eh. Fancy hashing. And a clever incentive scheme. That’s it.”

Later that week, I also found out you could mine Ethereum. I had two GTX 970s home at that time, so over the weekend I downloaded ETH miner, and tried to run the mining software.

But I didn’t. For whatever reason, I thought you had to download the entire blockchain(you don’t) at the size of 250 GB. My ISP had a bandwidth cap back then, and I didn’t want to pay more money and be crazy enough to download that much data. So I didn’t.

To this day, every time I see the ethminer folder with the last modified timestamp in 2017 on that hard drive, part of me cringes and let’s out a sigh. What if I questioned things a little more, and looked into it then? What if I actually said, fuck the bandwidth cap. Or what if I pickedf up the phone, and try to negotiate some kind of agreement with my ISP, so they waive the bandwidth cap as a one time courtesy that month? I mean, I spent so much time lowering my internet bill by having the representative get approval to match a competitor’s price point, so why didn’t I try that?

Around that same time frame, I also met up with my friends from high school. My friends were hardcore computer nerds who spent all their high school days at the computer lab. I asked my friends if any of them heard about bitcoin in 2010-2011. Them?

“Yeah we saw the white paper and laughed at how dumb it was.”

2019

In 2019, I went to a programmer’s retreat at the Recurse Center in NYC. There I met two people who’d become friends. One of them was an early investor in Ethereum and somehow made a significant amount of winnings to not look for a job for a long time. The other worked for a blockchain company, and was a huge blockchain enthusiast.

He asked me, “What do you not like about blockchain? Why are you not convinced?”

I don’t quite remember my answer then. It was probably something about how the financial system we have is already good enough.

He told me he was working on a blog post for programmers who are not interested in blockchain.

It ‘s here: https://jonsjournals.com/blog/2019-03-02-crypto-prices/

2021

One of my family members got interested in blockchain. They told me to make an early investment because an acquaintance was starting a blockchain. Told me there was a friends and family round. The acquaintance was one of the a smartest people I’ve heard of 2nd hand, so I told a leap of faith and invested.

Through the entire investment process, I created a Coinbase account, generated my own ETH address, and made my first transaction on chain.

Only when I made that first transaction did it all clicked.

  • No middleman.
  • Can create a bank account instantly.
  • Transaction set in stone, irreversible for all of time, out there for all the world to see.
  • Verify details of transaction from multiple sources on the internet.

This is when everything came together. It’s not just fancy hashing trickery or a clever design of incentives. or the 21st century’s equivalent of tulipmania. The way transactions are done through the blockchain - there are clear advantages. There’s definitely something there.

Over the entire year I’ve made a couple of transactions, did my research, and ultimately, I’m convinced that this is technology that will change the world for the better. It’s something that we absolutely need. It made me think deeply about what money is, and how this new technology is different or the same. Let me summarize the common arguments I’ve had before and a my thoughts on those common rebuttals.

What is money?

To me, money is a medium of exchange and store of value. Why is money valuable? If you look at your bank account, it’s just a number on a screen right?

What makes money valuable is that there’s this implicit expectation that someone else, government, entity, whoever, will take it from you in the future. In some sense, there is this element of procrastination involved and a perspective into the future.

Let’s say you have $200,000 in hard cold cash in the trunk in your basement. All of a sudden tomorrow, for whatever reason, everybody on the planet decides not to accept dollars anymore. Or the U.S. government says, sorry pal, we don’t accept paper bills anymore. Then the value of your stash of cash? Nothing.

So it’s critical for money to have people who are willing to take it. If they don’t, then it’s worthless.

Prior to blockchain technology, there had to be a central authority. Someone to make legitimate bills that are hard to copy. A central authority, due to lack of economic information, has to manually decide to reduce money in circulation, to keep the overall value of money constant in response to changing demand.

But with the advent of the Internet(unlimited access to information), plentiful computation(no longer requiring humans in the loop to manage money in circulation), all those assumptions that held true are no longer valid.

With that in mind, let’s address some common criticisms of blockchain technology.

1. We already have a financial system. There’s no need for another one

Have you interacted with our financial system, taking a look at how shitty it is? Wire transfers take 2-3 days. Absolutely ZERO transparency. You send money using traditional finance, and the money disappears from your bank account for at least a good 2-3 days, and then it shows up from the other side of the void. During this time, there is absolutely NO INFORMATION on your end. Doesn’t show up on the receiver side. All it says is…pending..

On the blockchain, anyone can look up your transaction, see what the status is on the pool waiting to be picked up. There’s absolutely zero transparency in such a current system.

We have no idea how it all works. It’s so opaque, code closed source. Why can’t all of this logic be open for all to see? The minute you change the property of a financial system from open to close, then you have to have almost all the necessary attributes of blockchain technology.

The other common argument that follows is:

2. Transactions are irreversible

Let’s take a look at the first page of Satoshi’s white paper:

“ Commerce on the Internet has come to rely almost exclusively on financial institutions serving as trusted third parties to process electronic payments. While the system works well enough for most transactions, it still suffers from the inherent weaknesses of the trust based model. Completely non-reversible transactions are not really possible, since financial institutions cannot avoid mediating disputes. The cost of mediation increases transaction costs, limiting the minimum practical transaction size and cutting off the possibility for small casual transactions, and there is a broader cost in the loss of ability to make non-reversible payments for nonreversible services. With the possibility of reversal, the need for trust spreads. Merchants must be wary of their customers, hassling them for more information than they would otherwise need. A certain percentage of fraud is accepted as unavoidable. These costs and payment uncertainties can be avoided in person by using physical currency, but no mechanism exists to make payments over a communications channel without a trusted party.”

Huh. So it seems like one of the main points for the invention of bitcoin is… irreversibility?

You easier to build reversibility on top of irreversibility, but not the other way around.

If you’re handing your someone $200 dollars bills to buy something, would you want this to always be reversible? Absolutely not. Once you hand in the cash for a good, then you go home, and take back the money. That’s just not how transactions are done..

Of course, there are transactions that you want to reverse. Credit card chargebacks are critical to commerce on the Internet. But why can’t we build this on an open platform instead? It’s also addressed in the original whitepaper…

3. You can send money to the wrong address and lose your money forever

This is a valid criticism. My stance on this is - user experience has not caught up to the technology yet. Unfortunately, the UI has not yet kept up to underlying tech. But all of this mistake can be avoided, if engineers implement a couple of steps, exactly the same as ones we have in traditional finance. Here are a couple:

  • Do a test transaction. You would never, ever send huge amounts of money without first doing a small transaction first, just to make sure you do it right.
  • Transfer apps should have more information about the address. The current balance, who has recently sent money, etc. If this information is displayed to the user before they commit, it will save a lot of heartache.

4. If you lose your private key, then you lose all your money

How is this different from the current banking system? If someone has my social security number, name, address, etc, there’s a good chance that they can spoof my bank account. If you write any electronic payment to your bank, you only need your routing number and bank account number. If you think about private keys as your social security number, and your public key as your bank account address, then leaking your social security number, and your bank account number, will either way result in catastrophic damage to your life as well.

In the United States, if things go south, the federal government guarantees to reimburse you up to $200,000. Of course there isn’t this kind of safety on the blockchain, but can we really rule that out? You pay the bank some kind of fee, or the right to store your money, who is insured by the federal government. And most of the time they have boatloads of earnings that are not going to you. I’m sure you can make a similar arrangement through another intermediary, or multiple intermediaries on blockchain platforms. This argument is not solid, but I believe the infrastructure and technology hasn’t yet caught up. Also, in real life, you don’t store life changing money in one bank account. The proverbial saying, “Don’t put all your eggs in one basket” also applies in crypto.

5. BUT IT CONSUMES SO MUCH ENERGY FOR NEEDLESS WORK. WE ARE KILLING THE EARTH

There are so many things in the world that consume so much energy, but do not provide a lot of utility to the world in terms of pure energy efficiency. I eat a lot of things but failed to make useful things. And I’m also killing the earth. Do I need to be gone? Should we get rid of everything that’s not adding value?

I’m not saying Proof of Work is great. We just haven’t figured out the right mechanism yet. It could be a different consensus protocol like proof of stake. It could be some a brand new scheme where instead of doing Proofs of Useless Work, we find a scheme or problem to get computers to do Proof of Useful Work that we can distribute to all powerful computers.

But to dismiss the entire paradigm for one single component is not good.

6. Decentralization is a myth

Argument: Core developers are the ones who control implementation details for blockchain platforms. They are the ones in authority. We’re swapping out one tyrant in favor of another.

The difference in cryptocurrency, from our current financial system, is if anyone doesn’t like how the blockchain is set up, they can take the codebase and start their own cryptocurrency. An escape hatch is always provided to this new modern financial platform.

But let’s suppose that it is indeed centralized. At the very least, it’s mostly a meritocracy, Anyone can write improvements, suggest fixes, and voice their opinions. Do we have that level of autonomy within our own financial systems without getting an Ivy League degree, or working up the corporate or government ladder promotion by promotion?

And if it really is centralized, in a society where everything is run by computers, would you choose to have hardcore computer tamers to run the financial system, or career bureaucrats? Who should dictate policy? Plato wrote(paraphrased), “The one who should rule is the one who desires it least.” Do programmers fit the description better, or career government officials who specialize in argument, campaigning, and persuasion?

Concluding Thoughts

Every human alive should have access to the Internet . To consume knowledge to their heart’s content. Then the next logical step is for people to be able to make bank account, and send money to whoever they want privately. Sending value to anyone anywhere should be a fundamental right in the Information Age.

There are people in 2021 who remain skeptical and unconvinced. I think it’s simply because they don’t understand the underlying technology. Take a look at this video of Bill Gates and David Letterman.

https://www.youtube.com/watch?v=gipL_CEw-fk

If Bill Gates was running a blockchain company and went on Letterman, how would the conversation change?

….

David Letterman: I think about this, and what about this blockchain thing? Do you know anything about that?

Blockchain Gates: Sure.

David Letterman: What the hell is that exactly?

Blockchain Gates: Well it’s become a place where people are publishing their financial statements. Everybody can have their own bank account in the open and you can make one instantaneously. Companies are there. With the latest financial information. It’s wild what’s going on! You can send money to people. It is the big new thing.

David Letterman: Yeah but you know it’s easy to criticize something you don’t fully understand, which is my position.

Blockchain Gates: Go ahead.

David Letterman: A couple of months ago there was like a big breakthrough announcement that on the internet or on some blockchain deal, they were going to send everyone a ton of money…airdrops? You could send money on the blockchain. And I just thought to myself - does… a wire transfer a bell? You know what I mean?

Blockchain Gates: There’s a difference. It’s not a huge difference.

The difference is that you can send money whenever you want, without a middleman, and you can see all your transactions, and also everyone else’s.

David Letterman: Right - oh I see so it’s stored in one of your memory deals.. And you can come back to it and …

Blockchain Gates: Exactly.

David Letterman: Do monthly financial statements ring a bell?

Blockchain Gates: Hahahahahahahahahahaha.

David Letterman: Yeah I just don’t know. What can you, just, knowing me, the little you know me now, what am I missing here? What do I need?

Blockchain Gates: Well, if you want to learn about digital art and the return on your money you keep there is a lot better…

David Letterman: I subscribe to two British magazines devoted entirely to art and I make a deposit to my 401k from my paycheck every two weeks. So now… now will the blockchain give me more than I’m getting that way?

Blockchain Gates: Oh, you can find other people who have the same unusual investing interests you do and….

Blockchain Gates: You mean the troubled loner gambling whale on the blockchain?

*laughter*

Installing Nvidia Drivers on Fedora 33 Issues

Here are some issues that I encountered installing Nvidia drivers on Fedora 33.

Nouveau drivers

Error:

The Nouveau kernel driver is currently in use by your system. This driver is incompatible with the NVIDIA driver, 
and must be disabled before proceeding.

Solution:

echo 'blacklist nouveau' >> /etc/modprobe.d/disable-nouveau.conf

Kernel Headers

Error:

Your kernel headers for kernel 5.10.21-200.fc33.x86_64 cannot be found at /lib/modules/5.10.21-200.fc33.x86_64/build or 
ln. /lib/modules/5.10.21-200.fc33.x86_64/source.

Solution

  648  sudo yum groupinstall 'Development Tools'
  649  sudo yum install kernel-devel

GCC Compiler Versioning Issues

Error

 ERROR: Failed to run `/usr/sbin/dkms build -m nvidia -v 465.27 -k 
  5.11.16-200.fc33.x86_64`:
  Kernel preparation unnecessary for this kernel.  Skipping... 
 
   Building module: rnel module:
   cleaning build area...
   'make' -j12 NV_EXCLUDE_BUILD_MODULES=''
   KERNEL_UNAME=5.11.16-200.fc33.x86_64 IGNORE_CC_MISMATCH=''                  
         modules...(bad exit status: 2)
  Error! Bad return status for module build on kernel:
  5.11.16-200.fc33.x86_64 (x86_64)
  Consult /var/lib/dkms/nvidia/465.27/build/make.log for more 
  information.

Solution

Make sure your version is downgraded to the one that is matching the one specified in make.log

After the install I believe you should reboot

Google Cloud - the bucket name is taken. Try another.

This is one of the bugs that I came across in Google Cloud Platform.

That bucket name is taken. Try another.

And then I go check the Google Cloud Platform and the dashboard, and I don’t see anything….

As of March 27, 2021, it seems like if you delete a project with a bucket, GCP won’t release the bucket name. It marks it as not having a permission issue. To resolve this, you have to run this command.

gsutil rm -r gs://linguasaga.com <-- linguasaga.com is the name of the bucket

Another word of advice - it seems like there are instances where the functionality of the UI is lagging behind the command line functionality. If there is a UI discrepancy, make sure you use the command line/SDK tools!

Site built with http://lanyon.getpoole.com/ template. Thanks to @mdo for the original template!