11
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
I finally have something to share in this thread!
I'm a system admin for HR systems and while programming is definitely not necessary to know, or do it have been extremely helpful for me the past two years. I'm the only person on my team who can do any programming at all, so when I do work on simple scripts, I get kudos for taking otherwise manual work and automating it.
One of my scripts that I have been iterating over for the past year is a python Selenium script (formally a PyAutoGUI script) that completes outstanding to-do items for managers across the entire company. In its current state, the script automatically kicks off at 9:00am every day, grabs the latest data file from my Outlook email, and uses a portable Firefox browser to login in as me and complete these items. At the end, it sends out a list of the remaining to-do items that couldn't be completed to a couple of users who previously completed the whole list of to-do items, throughout the day. Generally, I have been able to complete ~90% of the tasks, greatly reducing their workload. The problem that I have been facing is that the script:
Isn't written cleanly. I use two custom functions in its current state, the rest is ~400 lines of code that references tons of XPATHs for elements.
Can only be run off my PC. So, if I am not in the office, we have workers who have to complete this list manually while I am out of office. Or I can kick the script off on my vacation but that tends to suck.
Has very little error handling and tons of nested Try and Excepts. TimeoutError is the bane of my existence with this script and if I could figure out a good way to handle it, I would solve a major issue with this method.
Doesn't have a UI. It’s all just a .py file that requires python to run and the knowledge to troubleshoot.
For the past two days I started to rewrite this script and address some of these issues. I plan on taking a Leave of Absence from work next year and I want to pass some of these scripts to other users, so business won’t be impacted. Some of my planned changes are:
Create a UI for the script. This includes setting the folder where the file is located, entering an email distro list, setting the wait time for the page load, and any other setting that needs to be configured. This is mostly done so far using PySimpleGUI.
Break all the code up into different chunks through custom functions. Hopefully this will allow whoever inherits this code to be able to read the comments and purpose for each function and debug it in the appropriate parts. No function is overly complex, and everything is commented as cleanly as possible.
Write custom errors for the human error that will likely occur. This includes, selecting an empty/wrong folder, not indicating a correct setting in the UI, email errors, etc. Really, anything a non-tech user can encounter and know what to say to me if it errors out.
Package everything (script, python, dependencies, portable web browser) into an executable file so it the program can be double-clicked to run. I'm hoping this part goes smoothly but I think the biggest issue I will encounter is the portable web browser packaged into it. Due to company IT restrictions, I can only use portable firefox as the other browsers have disabled the ability to be run via web driver.
It will be interesting to me how it all turns out. I hope it works because it will mean more programming side projects for 2020 and extending out to the greater-HR group. Its easy to sell ROI on a simple script because the value being saved can be easily calculated, vs. how difficult it is to calculate ROI on other HR initiatives.
If anyone has any thoughts or advice, I would appreciate it and take it to heart! I’m a fairly novice programmer and still learning so pride won’t get in the way of any suggestions.
Side Note: I was about to post this last night in the last programming thread but didn’t think it appropriate to do so with the new one coming out less than 12 hours later.
Any thoughts about automatically migrating parent comments and their comment chains to the newest thread if a comment is made in one of the old threads or if a new parent comment was posted within 24 hours of a recurring topic?
Is there a reason you need a whole browser? Would it be possible to use something like a curl library to login, grab the session cookie from the response header etc, and basically just do everything through POST requests?
I wish I could use a curl library but my IT department blocks any sort of request originating from Python. This means it extremely difficult to do simple things like using requests or urllib to do tasks online. It's impossible to do something as simple as a "pip install x" as the firewall won't allow the request to go through to pypi. I end up having to download modules and their dependencies one by one. Combine that with disabling web drivers for already installed browsers, I have to use this terribly bloated approach.
Huh. That's crazy. I'm trying to wrap my head around this. The only way I can think of right now is they're using windows firewall rules to whitelist firefox.exe, outlook.exe, etc, and block everything else. They can't be detecting user agents since that wouldn't work over HTTPS. The block must be happening locally through a group policy of some sort. I wonder if this might be as simple as renaming python.exe to firefox.exe.
It's weird and incredibly frustrating!
The paranoid side of me believes IT monitors my network traffic because some things will work one day, then the next day it's blocked.
I believe this issue lies within their proxy server configuration, at least for using requests, urllib, or pip. The message I get when using those is the connection being forcibly closed by the remote host. My worst case scenario workaround is to connect to my mobile hotspot to bypass these restrictions if I have to.
Well surely they're monitoring your traffic, but it seems strange that they can tell the difference between requests from one application vs another. They can block you from contacting certain sites or protocols, but if say, you're making a HTTPS request on port 443 to google.com, they shouldn't be able to tell if you're doing it via firefox, chrome, or a python library.
The only way they could do that is by decrypting the packet, then inspecting the User-Agent string in the HTTP GET request. I suppose if they have a root certificate installed on your computer they could decrypt the packet, though that has far-reaching security implications, so I kinda doubt they're stripping off the SSL of every page you view the reencrypting it with their own cert. Even then, if you were using python to make that request, you could tell urllib or pycurl or whatever to forge a User-Agent to match firefox and as far as I can tell there'd be literally no possible way for a router/firewall to know the difference.
The only way they'd know what application is making the request is a local firewall rule on windows, on your computer itself, say, blocking everything except firefox.exe or outlook.exe etc from making requests.
I wonder how I could check that. I just tried changing my user agent in the requests module to my current chrome user agent and tried to get Google.
I encountered an SSL error, max retries exceeded caused by SSL Error(bad handshake, SysCallError 10054).
Sorry I'm on mobile as I don't post on Tildes from my workstation. This stuff is beyond my understanding though!
Hmm! That does sound like maybe they've got a root certificate set up and are decrypting and reencrypting stuff to do deep packet inspection.
I won't claim I'm an expert in this either. Maybe someone else can chime in. But what I'd try is in firefox, go to a SSL-enabled site, like https://google.com, then click the padlock icon in the address bar, click the
>arrow button, hit "more information", then "view certificate".Google's cert should look like this - assigned to google, mountain view california, and its root authority should be GlobalSign. If yours has the name of your business, or maybe the name of your firewall, idk, then what's happening is your company is able to decrypt all encrypted communications on your computer. And python somehow isn't using your company's certificate.
I'd assume if that's the case, you can download the certificate chain from the link on the firefox certificate info page, and pass it to urllib using the
cafileparameter.Again, I'm really not an expert in any of this, but that's what I got out of the SSL error anyway.
I just checked and verified that I do receive the same certificate you linked in Firefox. The mystery continues...
Sounds like a webfilter is installed on your work network, which is blocking these requests. If your IT department is reasonable, you might be able to get some exceptions coded into the webfilter if you submit a support ticket detailing your problem. Back when I was an admin, I helped some professors with the same kind of thing you're dealing with, which happened because of the webfilter being overly restrictive.
My main advice would be to get plenty of feedback from the people who will be using your new tool. You will want to ask them to try it out in a low-stress situation and watch what they do. I hope you have a good relationship with them so they tell you what they really think. Will you be training them?
It might be better to try it out with one feature that's working and add features after you learn more.
Definitely, and this is good advice! I have a very good relationship with this group and will be the one training them. I pretty much will be on call for them if anything happens, keeping my old scripts ready in case I have to redo parts of the program. My biggest goal is to make it in a way where they can set it and forget it as much as possible.
I know this isn't an entirely relevant comment for this thread, but what I'm discussing below is related to the software applications we're working on at the moment:
Far out. Starting a business, is emotionally, probably one of the hardest things I've ever done. My emotional resolve is better than it once was, but, hell, it's still like walking into a fire. Having the determination to put in 8+ hours of unpaid work a day, and somehow ignore the worries of "is this going to make enough money?", "will people want to use this?", "how do I price this?" is not an easy feat. I have broken down and cried about this a few more times than I care to admit—and yet, I keep working on this, because I know it's going to work. You gotta' believe in yourself, regardless of the odds, regardless of how you feel.
What's made me feel a bit better about this is I've recently got into a habit of breaking down our product/feature roadmap with my business partner, and timeboxing small, discrete components of functionality or utility within the application, setting an exact scope, expected duration to complete, and a due date. If I say "this will take me 4 hours", when I come to implementing this feature, I start a timer on my phone, and work until it's done. If I'm lucky, I beat the timer. Being able to literally check something off is a great feeling—you've walked another step on the 10,000 step journey to success.
When I don't feel like doing work on features, I'll work on spring cleaning. Adding tests, making UI improvements, etc.
I have recently finished a big project, si78c. This ended up being a really tough nut to crack, as I haven't reversed anything of that size before.
I am currently thinking about and hacking around on some possible followup projects. One current thread I am pulling is getting it to run on the web. Rather than going down the emscripten rabbit hole, I am exploring other simpler approaches.
I just finished my own client for the AI dungeon 2 game! It's just text RPG, but the story is generated by AI, not by human. I played it and it's really good! Sometimes, the story it generates is just awesome.
However, if you want to play it on computer, you have to use the google research colab notebook, which is awful for UX and it takes like 10 minutes to load; or their new iphone/android app. The app is great and works flawlessly, but I like typing on my computer.
So I
reverse engineeredinferred how to use their API and made CLI client! It actually works pretty well and I'm playing on it right now. And as a bonus, I learnt a lot about http requests and error handling in rust.So, I had an idea.
I want to combine a voice assistant with a game controller, to be able to speak out a novel, a letter, a screenplay or anything else and use the game controller to have better control of punctuation, tabs and special characters. Ideally this would be web-based, so that if you wanted to use it with a computer a mobile, or a VR headset, you could do that, but it would be nice to be able to use it with a smartpuck if there was the ability to do so. Where would you all recommend where I start?
So, like you'd have certain buttons on the controller mapped to period, question mark, etc?
Any reason for a game controller specifically? I can forsee a bunch of problems with compatibility with different devices, and having to map buttons with each one. I think your best bet would be if this device shows up as a bluetooth keyboard and sends the correct keycodes by default.
As such you'd be throwing out all the electronics inside the controller and replacing them with your own. At which point the controller itself is just a shell, and you can really use any shell you like. If you have access to a 3D printer you can make the device any shape you like.
To accomplish this, you'll want to buy an Arduino, a bluetooth module, some buttons or keyswitches, a battery and associated charging circuitry. Total price for parts should come in under $20, depending on what switches you use. It'll be a bit overwhelming at first if you've never built an electronic device or programmed in C, but it's totally doable, and easier than it might look.
The /r/MechanicalKeyboards community on reddit has some good resources on their wiki. There's a wealth of info there on building custom keyboards, maybe too much. Let me know if you have any specific questions. I've built things like this before using HM-10 BLE modules and cheap arduino nano clones off ebay.
I was thinking something like an Xbox controller or a one handed remote, hook it up to a phone/PC/device, and then map keyboard commands to it rather than gutting the inside and making it a hardware project, but a wirelessly enabled controller like a Stadia would be cool just to have it work with as many things as possible.
It's sorta difficult to remap buttons for every device you'd want to use this with. I have no idea how you'd do it with a Stadia controller, but to me that sounds even more difficult.
On android devices, you can remap controller buttons to keyboard buttons reliably if you have root access. Otherwise, there's apps like GameKeyboard+ that don't require root, but YMMV, a lot of people report issues getting it to work.
The most foolproof method that works on every device would be a bluetooth keyboard, or something that identifies itself as a bluetooth keyboard, and sends the correct keycodes for punctuation by default.