92 lines
6.9 KiB
HTML
92 lines
6.9 KiB
HTML
<head>
|
|
<title>Nicholas Orlowsky</title>
|
|
<link rel="stylesheet" href="/style.css">
|
|
<link rel="icon" type="image/x-icon" href="/favicon.ico">
|
|
</head>
|
|
<body>
|
|
<nav>
|
|
<a href="/">[ Home ]</a>
|
|
<a href="/blog.html">[ Blog ]</a>
|
|
<a href="/projects.html">[ Projects ]</a>
|
|
<a href="/extra.html">[ Extra ]</a>
|
|
<hr/>
|
|
</nav>
|
|
|
|
<h1>Side Project Log 7/12/2023</h1>
|
|
<p>This side project log covers work done from 4/29/2023 - 7/12/2023</p>
|
|
<p>This side project log is more than a bit late due to working a busy summer internship at <a href="https://futo.org">FUTO</a>!</p>
|
|
<h2 id="nws-container-deployment-service">NWS Container Deployment Service</h2>
|
|
<p>I have finally added SSL to NWS CDS. This was challenging, as it required handling ACME challenges and certificate
|
|
distribution across a set of geo-distributed Kubernetes clusters. I detail the complexities of this in a <a href="http://nickorlow.com/blog/ssl-in-nws-cds">previous blog
|
|
I wrote</a>. In order to implement auto-created/auto-renewing SSL, I implemented the below solution:</p>
|
|
<p><img src="/blog-images/NWS_SSL_Diagram.png" alt="Diagram of NWS SSL Architecture"></p>
|
|
<p>First, a user creates a request to add SSL to their NWS CDS service through the web UI which calls the NWS API (not pictured)</p>
|
|
<p>Then, the NWS API calls SSLiaison (in-house written software) which adds the domain to Caddy's list of domains. Caddy will then attempt to create
|
|
an SSL certificate from an ACME server (not pictured).</p>
|
|
<p>The ACME server will query NWS for the challenge response by requesting a file at <code>/.well-known/acme-challenge</code> on the
|
|
domain to be verified (this is the green arrows). </p>
|
|
<p>HAProxy will re-route these requests to NWS Hill Country, which is where the NWS Management Engine (NWSME) lives (this is
|
|
the orange arrows). <em>(NWSME controls what's deployed on each k8s cluster on NWS)</em></p>
|
|
<p>HAProxy in NWS Hill Country will then route this request to Caddy, which will solve the <a href="https://letsencrypt.org/docs/challenge-types/">http-01 challenge</a>, and then get the certificate
|
|
from the ACME server. Once it does this, it will write the certificate to a directory that is bind-mounted to both Caddy
|
|
and SSLiaison. </p>
|
|
<p>SSLiaison will detect this new file, parse it into a k8s manifest file, and then add it to our GitOps repo which is
|
|
hosted in GitHub.</p>
|
|
<p>From here, the certificate will be added to all the k8s clusters via Rancher Fleet.</p>
|
|
<hr>
|
|
<p>For next steps, I'd like to revise this solution such that it doesn't have a single point of failure.
|
|
Currently, if NWS Hill Country is down (which it is about 0.025% of the time), then SSL certificates
|
|
won't be able to be created or renewed. </p>
|
|
<p>To do this, I will have SSLiaison implement the ACME client specification so that it can create and respond do ACME HTTP challenges.
|
|
SSLiaison will run on NWS CDS (so that it's running on all of our k8s clusters and is HA) instead of running as a standalone docker container.
|
|
I'll have SSLiaison use some distributed database (probably CockroachDB) to store the HTTP challenges so that it doesn't matter
|
|
which k8s cluster the challenge request from the ACME server is routed to.</p>
|
|
<h2 id="next-steps-for-nws">Next Steps for NWS</h2>
|
|
<ul>
|
|
<li><strong>HA NWS Management Engine:</strong> Currently (as somewhat discussed above), the NWSME will go down when NWS Hill Country goes down. I'd like to
|
|
make it so that this isn't the case. This would likely just require that each NWS cluster runs its own instance of Rancher Fleet instead
|
|
of one central Rancher Fleet that manages all the clusters. It would also require the HA SSLiaison.</li>
|
|
</ul>
|
|
<ul>
|
|
<li><strong>IaC for Everything:</strong> Currently, all NWS CDS services are defined in yaml files in a git repo, however the underlying infrastructure that
|
|
it runs on is not. Ideally, I'd like every NWS machine to run Proxmox and then have Terraform & Ansible configs to define how to set up
|
|
vms on proxmox that will run the k8s clusters that support CDS services. This should eliminate my headaches of sshing into each machine
|
|
to apply config changes and make sure everything is standardized. It should also make the process of setting up new servers easy.</li>
|
|
</ul>
|
|
<ul>
|
|
<li><strong>Monitoring:</strong> I've been working on setting up monitoring on a lot of our services at my current internship using the TIG stack (Telegraf,
|
|
InfluxDB, and Grafana). Now that I've been exposed to the usefulness of having a bunch of metrics on hand, I think it would be nice to have
|
|
some dashboards setup for NWS to monitor speed, resource usage, uptime, and traffic. Doing this would also make it possible to expose resource usage in the
|
|
NWS dashboard.</li>
|
|
</ul>
|
|
<ul>
|
|
<li><p><strong>Enhanced Infrastructure:</strong> This is kind of a blanket one for things I want to do that don't fit into other categories. It includes making
|
|
hardware upgrades (mostly adding more storage), make management more accessible (such as Dell IDRAC's WebSerial), some load testing to
|
|
identify painpoints, run an NWS machine in a cloud VM so I can say it's cross cloud (although my friends have told me this is cheating at creating
|
|
my own cloud), and trying to figure out how to set up an Anycast network. I don't think I can setup an Anycast network without selling
|
|
a kidney first. Renting a /24 CIDR alone would be more than I want to spend on a side project. I may look into setting it up with ipv6 only,
|
|
however I'd still have to jump through hoops to get an Autonomous System number from an RIR. I'll probably write a whole blog about Anycast
|
|
in the coming weeks.</p>
|
|
</li>
|
|
<li><p><strong>Reduced External Dependence:</strong> The main goal of NWS is to have no external dependence. In theory, everything but core internet infrastructure
|
|
should </p>
|
|
</li>
|
|
</ul>
|
|
<h2 id="olney">Olney</h2>
|
|
<p><code>Rust, ActixWeb, PostgreSQL</code></p>
|
|
<p>Olney is a new project I am starting with my friend <a href="https://sridharnandigam.com/">Sridhar Nandigam</a>. It aims to make
|
|
tracking your job applications easier. Most of my friends either use spreadsheets or Trello to track their job applications, I
|
|
think that we can make something that's a bit better for the job. Some features I'd like to have are: resume version attached to
|
|
your application, job posting notifications from job boards such as <a href="github.com/pittcsc/summer2024-internships">pittcsc</a>, and watching
|
|
your email for emails from recruiters. Currently, I have part of the backend setup with basic CRUD operations. Now that I'm done with
|
|
the latest batch of NWS work, this is next on my list to work on.</p>
|
|
<hr>
|
|
<p><strong>These projects had minimal/no work done on them:</strong> RingGold, SQUIRREL</p>
|
|
|
|
<footer>
|
|
<hr />
|
|
<p style="margin-bottom: 0px;">Copyright © Nicholas Orlowsky 2023</p>
|
|
<p style="margin-top: 0px;">Hosting provided by <a href="https://nws.nickorlow.com">NWS</a></p>
|
|
</footer>
|
|
</body>
|