Coming to you from the mind of Neil Glick (The Brain), a good old fashion, home cooked technical blog regarding storage, virtualization, security, and now cloud and backups! All Opinions Are My Own.
Hi Friends,
You have to take a look at this really cool 3d printed magnetic gearbox (gear train) I recently printed. I created a video on how it works. What is so cool about it you ask? The magnets and carbon steel bolts create valleys instead of teeth. Besides it being really neat to fidget with, the gears won't chip or break if something gets stuck in them and will harmlessly skip to the next valley until the blockage is cleared.
You can download the files from Thingivese here.
Hi All,
Check it out, I'm on YouTube now!
Video One is a 3D printed tourbillon I made. All plastic except two wooden dowls from a toothpick. :-)
Video Two is a 3D printed gear box that I made. This is super cool, it uses magnets and bolts as the gears. This way no teeth are touching and it can be used when physical meshing of gears is not possible.
Hope you enjoy!
Neil
Hi Friends,
Building my Prusa i3 MKS3 from the kit has given me a new perspective on how precise these printers are. I did a bunch of printing and I noticed the prints were starting to get squished on the edges.
Its not the best picture, but you can see the extruder head is definitely hitting the print and if I took a side picture you'd see it was very smooshed.
Time to check the belt tension!!
Sure enough, both my Y and my X belts were loose.
Prusa has a whole page dedicated to this, but here's some quick tips.
For the X-Belt
1. Loosen these three screws, the bottom right screw acts as a pivot to tighten or loosen the x belt in small increments.
2. Either loosen or tighten this hidden screw, marked with the purple arrow. My x belt was loose, so I needed to tighten the screw (clockwise). Righty-Tighty, Lefty-Loosie. :-)
3. I noticed when I got the right tension and tightened the three screws again, the belt was too tight. So I loosened the three screws again and loosened the hidden screw just a bit. Tightened the three screws again and the tension was perfect on X.
For the Y Belt
Note: I needed to flip my printer on it's side, be very careful!
1. Loosen the screw on the bottom holding the belt to the bed.
2. Tighten or loosen the screw shown. In my case I needed to tighten it.
3. Check the tension with the Prusa app and when it's good, tighten the bed screw again.
After adjusting the belts, no more squishing!
Neil
Hi Friends,
I want to (re)introduce you to a new old hobby of mine 3D Printing. For those that know me, know that I've been doing this for about 10 years now and I just found a old blog of mine!
Do I still feel the same?
In 10 years 3D printing has changed a lot!! Printers have gotten cheaper, faster and the print quality is so good now. I decided to take the plunge and I got a Prusa i3 MK3S. Believe it or not I got the kit and built it myself! The kit comes in a big box full of boxes and it was a bit intimidating. I should have taken pictures, but I'm stupid.
I took it in steps and tried not to build the entire thing all at once. Remember you can purchase a fully built version from Prusa, but I'm insane and apparently like a challenge.
I built the chassis first, then the "hot-end" and finally put it all together with the belts and all. Prusa has a fantastic manual and they even include gummy bears too keep you going!
Why did I get the MK3S and not the MK4S? That's a great question. Truth is I purchased the printer and life just got in the way and it sat for a bit. No other excuses except I wasn't ready for the challenge and when I finally got off my back-side I realized I was a generation behind already. That's technology for you.
Don't get me wrong, the MK3S is an absolute beast and a workhorse and I've been able to print some amazing things with this printer, but it's not latest and greatest and that's okay for now.
Belt tension is always a bit of science-fiction and you'll find every recommendation known to, well everyone on the Internet. Prusa takes the guess work out of belt tension with their Prusa Belt Tuner. You pick your printer and pluck the belt and the app tells you if it's too tight or too loose. BIG help!!
I've got lots to say, but wanted to show you some of the things I've created. I've also gotten into CAD drawing for 3D printing using Onshape. Here are two of my creations:
2. Lucet
If you have a 3D printer, you can print them out now!
The happy face keychain is pretty obvious, but what's a lucet you ask? A lucet is a tool from the Viking and Medieval days that helped you braid things like rope, cord, etc.
Here's a link to wiki:
https://en.wikipedia.org/wiki/Lucet
I made a bunch of different versions and this seems to be the easiest to start making rope right away.
Check them out and let me know what you think!
Neil
Hi Friends,
Thanks for continuing to go with me on my ChromaDB journey.
If you haven't had a chance to read the first two parts, here they are:
Today I've got a great topic that started me on this ChromaDB, AI, LLM, Vector database journey. How do I backup and restore my AI database?
Humans are funny, we leap before we look and it tends to get us into trouble. Case in point, new application technology. I love it, you love it, we all love new toys! Think of backup and recovery as the batteries for your new toy. A lot of times we're so excited about the new toy we forget the boring essentials like batteries.
Unfortunately backup and recovery is often seen as a boring essential, until the data is needed and then you're the most important person in the world.
Inevitably a test application becomes production over night and now it's your responsibility to protect that data.
So let's get ahead of the curve with this new application data and figure out how to backup and recover your new application before it becomes production!
With ChromaDB you have a few options, you can:
1. Run memory resident.
2. Create a persistent data file.
3. Run in client/server mode.
I'm currently running my ChromaDB with persistence and writing to a data directory so if something happens all my data will be saved. If you don't specify what type of relational database you want to use for persistence, ChromaDB will use SQLite >3.35 as the default database.
For my first test I wanted to try ChromaDB out of the box, so I used SQLite. Make sure you create the filesystem first and when you let ChromaDB what client you're going to use, type:
persistentClient = chromadb.PersistentClient(path="/where_your_data_filesystem_is")
That's awesome, I've got my ChromaDB setup, it's persistent, I can query my data, but now what? There are a bunch of different ways to backup your SQLite database, but if you have Veritas NetBackup it's SUPER simple to integrate this new technology into your enterprise backup and restore technology.
The cool thing about this is the SQLite agent is already built right into the NetBackup client software and has been since around 10.2.
Let me guide you through the process.
If you haven't downloaded and installed the Veritas NetBackup Client software on your ChromaDB box, you'll need to do that now. You can download the Veritas software from HERE.
1. Log into the NetBackup WebUI and navigate to Protection > Policies and click on the +Add button.
Hi Friends,
Even though I complained there were no good gadgets, I had to remind myself of some of the cool stuff that IS new and awesome.
Where have you been all my life?! I love the Nintendo Switch, it is an awesome game system. I've always enjoyed portable handheld gadgets, from the Walkman to the Gameboy. Something about having all your cool stuff with you at any time has always been appealing and the Nintendo Switch is the marriage of all the benefits of a big console in your hands!! Oh yeah, did I mention it also has a touch screen too? I know it's not new, in fact it's been out since 2017, but it's just so cool I had to be on the list.
This little guy is AWESOME! If you haven't seen him yet you have to go check him out. I've been a sucker for robots since I was a kid. Something about a super smart buddy that wouldn't judge you, made you laugh, would protect you and would always be by your side sounded pretty cool as a kid. EMO is a a desktop robot pet/friend. He enjoys talking with you, wandering your desk, dancing, singing, playing music and pats to the head. He's a pretty cool little dude! Living AI even has a new robot coming out soon called Aibi. I can't wait!!!
This is a neat little game system, it's very clever and I've never seen anything like it. It has the D-Pad and A & B, but also something very different, a crank. The crank is another controller on the device and it makes games very unique. Panic gives you a bunch of games when you get your system and even the way you get them is very clever. You can even develop games for it and purchase games created by a pretty large online community.
Hi Friends,
I got PIP to install!
I've been doing a bunch of research and wanted to give credit to some great blogs!!
Hasini Madushika's blog:
https://medium.com/@hasinivijayarathna/creating-a-vector-database-using-chroma-956b1d84aca3
Fantastic overview on how to setup your first ChromaDB and create a searcheable index of books and authors.
Michael Wornow's blog:
https://michaelwornow.net/2023/12/31/chromadb-demo
Great ChromadB overview with pros and cons as well as a great section on cosine similarity vs. distance.
Milana Shkhanukova's blog:
https://medium.com/@milana.shxanukova15/cosine-distance-and-cosine-similarity-a5da0e4d9ded
Fantastic job explaining in more detail what is cosine distance and how it's different from similarity.
Harrison Hoffman's blog:
https://realpython.com/chromadb-vector-database/
Another great blog on ChromaDB foundations as well as lots of information on vector similarity.
Who knew physics would actually be useful?! Yeah yeah, Isaac Newton did in 1687.
Now let's get rolling!
1. Let's make sure Python3 is working:
#python3 -V
Python 3.10.12
WOOT! And just to let you know, I'm using Ubuntu 22.04.2.
2. And to install the ever illusive PIP.
#sudo apt install python3-pip -y
#pip -V
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
ALRIGHT!
Now let's create a vector database!
1. The first thing I'm going to do is create a place to store my data.
mkdir /data
I did all this in a bash shell, but you can create a Python script and run it in there too if you'd like.
2. Run Python so you can run the code.
#python3
3. Import ChromaDB into Python.
import chromadb
Now here comes the fun part. Do you want your database to only run memory resident or do you want it to save some place? Kinda depends on your needs, the space you have, etc. But I don't want all my work to go poof, so I'm going to save it to disk.
4. Use this if you just want memory resident.
chroma_client = chromadb.Client()
OR
4. Use this if you want to save your database somewhere.
persistentClient = chromadb.PersistentClient(path="/data")
I'm going to save my data to filesystem called "data", but you can save your data anywhere you'd like.
What's really cool about this is ChromaDB saves this data into a SQLite3 database for you. If you're having trouble with it, take a look at the troubleshooting section on the Chroma website.(https://docs.trychroma.com/troubleshooting).
5. This is the really cool part! Next we're creating our collection. Here we're going to give it a name and you'll notice a geometry and physics term you probably hoped would never haunt you again, COSINE! What's going on here is we're telling the database to find words that have different vectors. If the vectors are different, they probably mean the opposite. If the vectors are facing the same directions, it's more likely they mean similar things. In Michael Warnow's blog he shows you how to find the similarity instead of the difference if you want to do that instead.
books_collection = persistentClient.create_collection( name="books",
metadata={"hnsw:space": "cosine"}
)
6. Next thing we do is add data to our books collection. I'm going to use Hasini Madushika book collection because her book titles do a great job showing how the cosine feature works. There's a lot going on here, but I think I'll break this down further in another blog about embeddings and such.
books_collection.add(
documents=[
"The Enigma Code", "Decoding Secrets", "Whispers of Intrigue", "Conundrums and Clues",
"The Puzzle Master", "Mysterious Ciphers", "The Labyrinth of Enigmas", "Cryptic Chronicles",
"Puzzled Minds", "Secrets Unveiled", "Echoes of Eternity", "Time's Embrace",
"Chronicles of Eternity", "Eternal Moments", "Timeless Whispers", "Infinity's Tapestry",
"Temporal Odyssey", "Endless Hours", "The Time Weaver", "Eternal Sands",
"The Silent Symphony", "Whispers of Silence", "Silent Serenade", "The Sound of Quiet",
"Quiet Harmony", "Harmony in Silence", "Muted Melodies", "Serenity's Echo",
"The Tranquil Note", "Echoes of Quietude"
],
metadatas=[{"author":"Alan Cipher", "price":"$19.99"},{"author":"Olivia Mystery", "price":"$18.95"},
{"author":"James Riddle", "price":"$21.50"},{"author":"Emma Puzzler", "price":"$22.99"},
{"author":"Alex Brainteaser", "price":"$20.75"},{"author":"Victoria Enigma", "price":"$23.45"},
{"author":"Samuel Conundrum", "price":"$24.80"},{"author":"Grace Enigma", "price":"$19.25"},
{"author":"Daniel Riddle", "price":"$17.99"},{"author":"Amanda Mystery", "price":"$21.00"},
{"author":"Robert Timeless", "price":"$25.50"},{"author":"Sarah Infinity", "price":"$26.75"},
{"author":"Michael Eternal", "price":"$24.99"},{"author":"Emily Timekeeper", "price":"$23.20"},
{"author":"Christopher Infinity", "price":"$22.45"},{"author":"Jessica Forever", "price":"$27.30"},
{"author":"Nicholas Timeless", "price":"$28.50"},{"author":"Laura Infinity", "price":"$26.00"},
{"author":"Benjamin Chronos", "price":"$24.95"},{"author":"Rachel Timebound", "price":"$25.75"},
{"author":"William Hush", "price":"$18.50"},{"author":"Sophia Mute", "price":"$17.75"},
{"author":"Oliver Quietude", "price":"$19.20"},{"author":"Isabella Hushington", "price":"$20.15"},
{"author":"Matthew Serene", "price":"$18.99"},{"author":"Emily Tranquil", "price":"$21.50"},
{"author":"Christopher Hushwell", "price":"$22.75"},{"author":"Grace Silentheart", "price":"$19.95"},
{"author":"Daniel Peaceful", "price":"$23.00"},{"author":"Victoria Hushed", "price":"$20.80"}
],
ids=["id1", "id2", "id3", "id4", "id5", "id6", "id7", "id8", "id9", "id10", "id11", "id12", "id13", "id14", "id15",
"id16", "id17", "id18", "id19", "id20", "id21", "id22", "id23", "id24", "id25", "id26", "id27", "id28", "id29", "id30"
]
)
7. Now that we've got data in our database we can query it!
results = books_collection.query(
query_texts=["Eternity", "Puzzle"],
n_results=5
)
print(results)
{'ids': [['id11', 'id13', 'id14', 'id18', 'id20'], ['id5', 'id9', 'id1', 'id4', 'id19']], 'distances': [[0.33300162331174143, 0.378593976226286, 0.42385445272206546, 0.45082819934444696, 0.5292349798068949], [0.44286587397375554, 0.5532287339143328, 0.5986420831809907, 0.6688832557674483, 0.6887007582505835]], 'metadatas': [[{'author': 'Robert Timeless', 'price': '$25.50'}, {'author': 'Michael Eternal', 'price': '$24.99'}, {'author': 'Emily Timekeeper', 'price': '$23.20'}, {'author': 'Laura Infinity', 'price': '$26.00'}, {'author': 'Rachel Timebound', 'price': '$25.75'}], [{'author': 'Alex Brainteaser', 'price': '$20.75'}, {'author': 'Daniel Riddle', 'price': '$17.99'}, {'author': 'Alan Cipher', 'price': '$19.99'}, {'author': 'Emma Puzzler', 'price': '$22.99'}, {'author': 'Benjamin Chronos', 'price': '$24.95'}]], 'embeddings': None, 'documents': [['Echoes of Eternity', 'Chronicles of Eternity', 'Eternal Moments', 'Endless Hours', 'Eternal Sands'], ['The Puzzle Master', 'Puzzled Minds', 'The Enigma Code', 'Conundrums and Clues', 'The Time Weaver']], 'uris': None, 'data': None}
8. Let's do a couple of other queries using our cosine vector.
print("results for 'Eternity':", results["documents"][0])
print("results for 'Puzzle':", results["documents"][1])
results for 'Eternity': ['Echoes of Eternity', 'Chronicles of Eternity', 'Eternal Moments', 'Endless Hours', 'Eternal Sands']
results for 'Puzzle': ['The Puzzle Master', 'Puzzled Minds', 'The Enigma Code', 'Conundrums and Clues', 'The Time Weaver']
This is really cool, notice the key words are Eternity and Puzzle. The query find the exact word, but also words with a similar meaning. They may not have the same magnitude, but isn't that cool?!?!
Until Next Time!
Neil
Hi Friends,
I was going through some of my old blogs and I came across
Hi Friends,
What's the first thing that comes to your mind when I say Artificial Intelligence(AI)?
HAL 9000
Computers are basically really powerful calculators and they do a fantastic job crunching numbers. The folks that make these databases feed the AI tons of text, images and writing examples that are translated into numbers. The LLM starts to see patterns in the text and images and begins playing, What's The Next Word game.
For example we can start something like, "A cat is a <blank>." and we go from there. A cat is an animal. A cat is an animal with four legs. A cat is an animal with four legs and whiskers. And on and on. The more information you feed your LLM the better it will be at answering your questions, but you have to remember that your LLM will only be as smart as it's data AND the questions it's being asked.
Soooooo I decided it was time for me to install one of these REALLY cool databases and find out how they work and how I'm going to back it up. I decided to go with ChromaDB, it's open-source and look how easy the install instructions are:
pip install chromadb
That's it?! Well sheesh, I can do that! Wait, what's PIP? Isn't it that thing you get put on when you're doing a bad job at work?
Will Neil be able to figure out what the mysterious PIP tool is? Can he get ChromaDB installed on his Ubuntu box?? All these questions and more will be answered in the next chapter of Me and ChromaDB!!
Hi Friends,
More Veritas Alta Recovery Vault news! I'll go into a lot more detail about the technology, but wanted to showcase two new documents and certifications for Alta Recovery Vault. Veritas Alta Recovery Vault and MSDP-C tech typically uses HTTPS to communicate with cloud vendors over port 443.
If you'd prefer to have a direct connection with either Azure or AWS you can use their connection tech. For Azure it's called ExpressRoute and for AWS it's called Direct Connect.
Why would you want a direct connection to your cloud service provider (CSP)? There are tons of reasons but a few could be speed, security, more speed. At Veritas we purchased the connections to the two CSPs and tested the performance with and without the dedicated pipe. How did it perform? Take a look at these two docs for all the gory techy details!
Alta Recovery Vault ExpressRoute Overview
Recovery Vault Direct Connect
Neil
Hi Friends,
I've been away for a bit heads down working on some really cool backup and recovery stuff! In particular I'm working on a product called Veritas Alta Recovery Vault which is built right into Veritas NetBackup. Alta Recovery Vault uses MSDP-C magic and connects you to Azure or AWS (your choice) storage that you can use as backups!
The super cool thing about the Alta Recovery Vault is it's immutable and can't be expired or overwritten in the event a threat actor gets access to your backup system. This is huge since threat actors are not just locking up networks and computers anymore, they're also expiring backups to ensure you have nothing to go back to once your network is infected and/or locked.
Are you thinking of transitioning to the cloud or already have backups on one cloud vendor and want to have backups on a second vendor? Alta Recovery Vault has you covered in all of those scenarios and it's also SUPER easy to setup!
Check out my new video on how to setup Alta Recovery Vault immutable storage here:
https://youtu.be/r0emXtBHKMM?si=uBaP6AzEnqmO-IVi
and then check out this Alta Recovery Vault video where I go into the backup system, take a backup using traditional non-immutable storage and then one on immutable storage. I go into the catalog and delete the non-immutable image, but can't delete the immutable copy on Alta Recovery Vault!
https://youtu.be/DaVvfo8qlKs?si=IF2oJlMFoaMTKgGd
I've got lots more to share, so here are a few more helpful links:
Alta Recovery Vault Security Profile
Alta Recovery Vault Deployment Guide
https://www.veritas.com/support/en_US/doc/VeritasAltaRecoveryVaultGuide
Alta Recovery Vault Deployment Demo Azure
https://www.youtube.com/watch?v=e_cR89y7Drk
Alta Recovery Vault Deployment Demo AWS
https://www.youtube.com/watch?v=oxMoS1E2xKI
Alta Recovery Vault Image Sharing
Alta Recovery Vault Direct Connect
Alta Recovery Vault ExpressRoute Overview
Until Next Time My Friends!
Neil
Hi Friends,
I'm creating a new NetBackup environment and I installed the Primary (Master) server and I noticed a question. Would you like to add Media servers? Well no, I haven't installed my Primary yet... Right?
So after I installed my Primary, I began installing the NetBackup software on my Media servers. You tell NetBackup it's NOT a Primary, you give the installer the name of the Primary and away you go!
After everything installed, I noticed my new Media servers showed up in the Hosts tab in the Web GUI, but when I tried to add one of my new Media servers as a Storage server, they weren't showing up... Hmmmm.....
To you NetBackup veterans, you're going to laugh at me, but for the life of me I couldn't figure out how to let the Primary server know this new server was a Media server.
From what I can see, there are a couple ways to do this. I'm sure there's lots of other ways, so let me know in the comments! :-)
1. Edit the bp.conf located in /usr/openv/netbackup
2. Add the machine in the Java GUI using jnbSA in /usr/openv/netbackup/bin
=========
Let's tackle number one first. On the Primary server go to /usr/openv/netbackup and edit the bp.conf file.
Here's what mine looks like:
SERVER = the_name_of_your_primary_server
CLIENT_NAME = the_name_of_your_primary_server
CONNECT_OPTIONS = localhost 1 0 2
EMMSERVER = the_name_of_your_primary_server
VXDBMS_NB_DATA = /usr/openv/db/data
OPS_CENTER_SERVER_NAME = the_name_of_your_primary_server (unless your Ops Center isn't the primary)
MEDIA_SERVER = the_name_of_your_new_media_server
SERVICE_USER = the_user_you_selected
WEBSVC_GROUP = nbwebgrp
WEBSVC_USER = nbwebsvc
TELEMETRY_UPLOAD = YES
The MEDIA_SERVER entry should not exist and this will need to be added.
After you add this line I've seen sometimes having to restart netbackup on the Primary, sometimes on the Media, but for me it eventually popped up as an asset and I could use as a Storage Server.
========
The second way involves opening up the Java GUI (jnbSA) under /usr/openv/netbackup/bin. When the GUI opens, click on Host Properties > Master Servers.
On the right hand side, right click on your Primary/Master server and select Properties.
Then click on Servers on the left hand side and select Media Servers on the right hand side. From here, you can add your Media servers, which will add the server(s) to your bp.conf file.
Check your bp.conf file again and make sure the Media server was added.
As I said earlier, I've seen recommendations to restart NetBackup on your Primary server, but for me the newly added Media server popped up as a Storage server in the Web GUI.
And that should be it! You should be able to use your new Media server(s).
Until Next Time,
Neil
