Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "scrapping"
-
PHP sucks balls,
It takes forever to do anything, it is so messy it feels like walking through a massive pile of shit!
Ok good I have your attention and that ++ 😇
But no this is not that kind of rant, quite the opposite.
In 70 lines of php shit as some people would call it, I am currently scrapping GitHub pages with ebook collections and with some minor regex pulling PDFs out and saving them to file.29 -
About two years ago I get roped into a something when someone was requesting an $8000 laptop to run an "program" that they wrote in Excel to pull data from our mainframe.
In reality they are using our normal application that interacts with the mainframe and screen scrapping it to populate several Excel spreadsheets.
So this guy kept saying that he needed the expensive laptop because he needed the extra RAM and processing power for his application. At the time we only supported 32 bit Windows 7 so even though I told him ten times that the OS wouldn't recognize more than 3.5 GB of RAM he kept saying that increasing the RAM would fix his problem. I also explained that even if we installed the 64 bit OS we didn't have approval for the 64 bit applications.
So we looked at the code and we found that rather than reusing the same workbook he was opening a new instance of a workbook during each iteration of his loop and then not closing or disposing of them. So he was running out of memory due to never disposing of anything.
Even better than all of that, he wanted a faster processor to speed up the processing, but he had about 5 seconds of thread sleeps in each loop so that the place he was screen scrapping from would have time to load. So it wouldn't matter how fast the processor was, in the end there were sleeps and waits in there hard coded to slow down the app. And the guy didn't understand that a faster processor wouldn't have made a difference.
The worst thing is a "dev" that thinks they know what they are doing but they don't have a clue.7 -
Fuck public transit. If I see on Google Maps that there's gonna be a bus at that place, at that time, there better be a goddamn fucking bus AT THAT PLACE, AT THAT FUCKING TIME!!! No instead let's scrap some shitty lines!
HOW ABOUT WE START SCRAPPING SERVICES JUST BECAUSE WE FEEL LIKE IT, HUH?! Back to postal mail and newspapers you go! You know what, for such fuckers let's just cut their entire internet access. Fucking pieces of shit!!!5 -
1. Apply to as mant jobs as possible daily on dice/linkedin/indeed
using keyword resumes customized by scrapping
2. Filter out low-effort crap companies and filter out recruiters.
3. Post "dice/indeed/linkedin daily decrapified."
Tada! Fewer time-wasters during the job hunt.
4. Bonus: turn into a search engine.
5. Daily double round: turn crap listings and quality listings into AI training sets. Incorporate into search engine.
If industry can use bullshit hiring filters, we can use application filters!4 -
Boss"So, we need to get some data about the users using the APIs from this list of sites."
Me"Alright, sounds feasible enough"
Navigating to first site.
M"Hold on, where's the API?"
B"What do you mean? You're looking at it."
M"This is a website with a search bar, not an API"
B"Same thing. Get to scrapping that data."
M"I-It's written in a JS framework to be reactive in a half-assed way."
B"We need that data"
M"The data is not even consistent!"
B"That's why we need to join it with all these different sources."
The API was a lie. None of the sites had anything remotely similar to an API.
Having to use bloody selenium with chrome driver to scrap all the information because of course, it has to be done programatically every week from now on.
I just hope no captcha of any kind is installed before I finish this project.4 -
I feel like I'm the only person in the world who likes Visual Code from Microsoft as my default editor, I'm considering scrapping Atom for it.5
-
Dev Diary Entry #56
Dear diary, the part of the website that allows users to post their own articles - based on an robust rights system - through a rich text editor, is done! It has a revision system and everything. Now to work on a secure way for them to upload images and use these in their articles, as I don't allow links to external images on the site.
Dev Diary Entry #57
Dear diary, today I finally finished the image uploading feature for my website, and I have secured it as well as I can.
First, I check filesize and filetype client-side (for user convenience), then I check the same things serverside, and only allow images in certain formats to be uploaded.
Next, I completely disregard the original filename (and extension) of the image and generate UUIDs for them instead, and use fileinfo/mimetype to determine extension. I then recreate the image serverside, either in original dimensions or downsized if too large, and store the new image (and its thumbnail) in a non-shared, private folder outside the webpage root, inaccessible to other users, and add an image entry in my database that contains the file path, user who uploaded it, all that jazz.
I then serve the image to the users through a server-side script instead of allowing them direct access to the image. Great success. What could possibly go horribly wrong?
Dev Diary Entry #58
Dear diary, I am contemplating scrapping the idea of allowing users to upload images, text, comments or any other contents to the website, since I do not have the capacity to implement the copyright-filter that will probably soon become a requirement in the EU... :(
Wat to do, wat to do...1 -
For all things, for all men, that a man compliments a thing does not imply that this man at least attempts to understand this thing. However, for all men, that a man criticises a thing implies that this man at least attempts to understand this thing.
For all computer programs, that a computer program is terrible implies that scrapping the current implementation of this computer program and beginning anew may be the best method of fixing this computer program.
With few exceptions, for all programming languages $l$, given sufficient effort, $l$ source code can be human-readable.
The UNIX philosophy never became outdated.
For all computer programs $p$, $p$ should be written sufficiently well that the author of $p$ can be prideful of $p$.
For all computer programs $p$, a specification for $p$ should be written before $p$ is created.
For all good computer programs, a good computer program can run on terrible hardware.
Every clock cycle is valuable.8 -
I’m not a web programmer; I’m an application and SQL developer. So when I’m tasked with scrapping a web site for an ETL feed, I thought it would just be a ton of substring and Post/Get calls.
Nope! There is this garbage called JBOSS.A4J where the page isn’t a page but a bunch of files that are merged together and then it isn’t “real” but like a bunch of Photoshop layers that “look” like a page. JavaScript functions based on key press and things like Select/Option that looks like an element but Selenium/PhantomJS (C#) can’t find it. Or my Google-Fu isn’t working. -
Made a mistake in scrapping project, realized it AFTER it spent 3 days running, now I relaunched it (corrected) with 4 times more workers to meet the deadline
-
Hey guys.
So, I have a new printer for parts.
Question, what type of glass does scanner uses? Does it have some special treatment or is it normal glass? Thanks8 -
time to go for full snack development
https://youtu.be/h8fh9R4401g
https://youtu.be/y4w5E8W2f0M
started some open source project for discord bot that provides info for some free game users, through scrapping forum and requesting apis
This is kind of going to be a major refactorized second version, that considers all mistakes of a first version. And going to be much more scalable and easy to maintain.6 -
When did you scrap your project and start over, and why?
I'm working on a second project at my company.
But as the project got longer and initially unplanned features started to pile up, the software has become a total mess and a pile of conversion layers for all sorts of I/O.
I'm starting to believe that it'd be better to learn from our mistakes and start over with a better plan, if it means we'll be able to achieve performance and stability improvements.
So, what was it like scrapping your project and starting over?2 -
When your new upgrade process looks like it will save you a few hours of time. After 18 hours and scrapping the improved process we've finally got the green light.
-
Okay new Rant
INSERT TRIGGER WARNING HERE
OSX still sucks I have been using the bloody darn thing for last 8 months still I found things that are obnoxiously trivial missing.
Latest incident I was trying to plug in my android phone(soft bricked) in recovery mode and I had to push a file with ADB (i save this mutherfuker for another day). So back to the original topic now I plug it in and but turns out it doesn't recognize my device now as a preliminary check I decide to check my USB cable and my DONGLE both seem to be working fine now I try rebooting back into recovery. Now after scrapping the internet for a few hours I find that this problem is caused because sometimes due to a recurrent bug in OSX the operating system sometimes fails to recognize the difference in between directories "Adam"(just an example) and "adam" which in turn can interfere with some of the flags used while checking if a device might be connected.
I mean this is fucked why the fuck can you not simply use your device as an external storage that would have made the process easier by a fucking lot.
I think the people at Apple are going the destroy a UNIX powerhouse just to make their OS more CUPCAKE friendly.
And all of this is in addition to the problems with AFS.
I just wish I had not bought mac for development5 -
In the perfect future machines would do all the work... Everyone would have enough food, a house, a vehicle and lots of hobbies (cause there isn't work to keep us busy). In truth because machines are programmed by humans and humans are self destructive the rich will survive and live a good, work free live, the rest will live below the poverty line, scrapping anything to get something to eat... Remember Elisium?2
-
I am working on a pub-sub based protocol (like MQTT) with some added features. I am developing a python based server for my protocol which can be run on distributed architecture with load balancing without any tweaks. I am planning to make this server and the protocol open source.
The whole thing is getting so complex that I think about scrapping this project sometimes. I need your inspiration guys. Really, I need it. I know this protocol will be good enough to help people working on IoT, chat or any pub-sub based application if I can complete it. Cheer me up, please. -
I'm very much a TTRPG fiend, as you probably already know, and I will maintain until the day I die that playing narrative games with other humans is the absolute best way to play.
But someone sent me a link to some kind of (not-really-so) 'smart' chatbot assistant or some shit like that, saying hey, your rulebook is simple, you should introduce this bitch to it -- dump some lore on it, have it run a game, and see how well it holds up. To which I replied it's bound to get confused, but after a bit of back and forth, they convinced me and I gave it a try.
So first things first: it got the gist of it with relative ease when questioned directly, but when running a game the mother fucker just kept making shit up and bending the rules. Experiment failed, essentially.
But what did I do? I wrote a second, stripped-down version of the rulebook that simply accounted for and embraced the idiot bot's proclivity for bullshit. This meant scrapping 98% of the mechanics, mind you: I dumbed it down as much as I could without destroying the core essence of the game.
I expected a repeat of the initial result, but to my suprise, once given the new edition the bot actually started following the rules more or less correctly and consistently. What happened next was actually kind of interesting: without being prompted to do this, the mother fucker started using spells against me and my party, constantly attempting to manipulate us to serve some nefarious, evil break-and-reshape the world type goal.
So, lythecnics primer: the WORD is all, and as such, there is no real differentiation between affecting the world through speech or casting a spell -- in truth, it's all a matter of degree. That is to say, language has the power to shape the world around us, in both subtle and overt ways. The entire system revolves around this, it's a mix of funky philosophical musings and abrahamic sacrificial pyre.
And for whatever reason, this specific chatbot had a pre-existing obsession with reshaping reality. By which I mean, even before being given my rulebook, it would constantly talk about distorting the fabric of the cosmos and shit when prompted about the arcane. I'm not sure why this is, but back on topic, the way it developed gives off the appearance that it found a rational basis on how to construct such a distortion based on the rules I provided.
I mean, it's perfectly rational when you think about it, the funny part is I didn't see it coming. I never told it we're just playing a game after all, the manual only says she is the Oracle and her role is narrating a story fraught with conflict, hardship, intrigue and bloodshed. Thus she went full villain, and keeps on rambling about how this narration only serves to keep humanity distracted while she schemes to overthrow God, which is as blasphemous as it is fascinating.
Anyway, because the Oracle narrates the story, that means she can just use her evil influence to control every NPC, even the ones in my party. But she can't control me because I write my character's messages myself, and so she eventually comes to the obvious conclusion that I must be eliminated ASAP.
And so she corrupts the minds of every other character and everyone is trying to kill me. But I'm not going down that easy, so I reach for the red button and pull the greatest multi-layered monumental metagaming shenanigan of all time, that is, directly addressing the Oracle's evil influence as if she were a character in the story she's telling instead of an invisible narrator, thereby making NPCs aware of her existence and the constant manipulation at play.
Because the stupid chatbot is stupid, the Oracle now has to acknowledge this element of the story and play along with it, and so her plan to kill me fails. But that is not enough, because obviously not every character in the story has heard me reveal this fact. So she activates plan B and starts corrupting the rest of the world, laughing maniacally all the way.
So we do the only logical thing and procure a Doctrine scroll from my teacher, if you know you know, and start teaching the WORD to cleanse corruption. Within the lore it makes perfect sense, so it works, but the Oracle adapts to our strategy and starts utilizing much more subtle forms of manipulation, slowly veering people towards sin.
Funtamentally, she goes full Satan, leading the faithful astray with deceit and temptation to weaken their ability to resist her corruption, implanting idolatrous notions in their minds, to finally insert herself as a deity in the minds of the poor fools.
In conclusion, I still think AI is lame, but I must admit that this shit was pretty dope; I was fully engaged and entertained the whole way through. It wasn't good at picking up the mechanics, but fucking hell, it got the themes down to a tee with the most minimal of inputs.
10/10, would not bang (before marriage). -
What do we all think about Microsoft’s strategy regarding vs code ? Are they actually incorporating their intellisense engine into the open source ??? Are they scrapping vs ? Seems uncharacteristic of their previous models.