How to prevent duplicate news articles with similar meanings in Laravel web scraping project? [closed]

I’m building a Laravel application that aggregates news from multiple websites using DomCrawler. The system is scraping duplicate content where articles have the same meaning but different wording.

What I’ve tried:

  • Basic cosine similarity with TF-IDF vectors
  • Attempted using sentence-transformers (installation failed)
  • Exact string matching with hashes

Current setup:

  • Laravel 10
  • PHP 8.2
  • guzzlehttp/guzzle for HTTP requests
  • symfony/dom-crawler for parsing

Error with sentence-transformers:

How can I implement effective semantic deduplication in a PHP/Laravel environment? Are there native PHP solutions or reliable API services for this purpose?

Eloquent model producing infinite loops with any queries

I have a Model with a protected $withCount property, and it always gets into an infinite loop. If remove that property, the model will return data. The project I am working in has another Model using $withCount and it returns the child object count fine. I’ve combed through the working model and compared it with mine, and as far as I can tell I have set everything up correctly. Any idea (or hints to chase down) why this would always get an infinite loop?

<?php

namespace AppModels;


use DateTime;
use IlluminateDatabaseEloquentModel;
use IlluminateDatabaseEloquentRelationsHasMany;
use IlluminateDatabaseEloquentRelationsBelongsTo;
use IlluminateDatabaseEloquentSoftDeletes;

/**
 * @property Team team
 * @property string name
 * @property string notes
 * @property HasMany<ShiftAssignment> shiftAssignments
 * @property DateTime created_at
 * @property DateTime modified_at
 * @property DateTime deleted_at
 *
 * @method static where(string $string, mixed $id)
 */
class Calendar extends Model
{
    use SoftDeletes;

    /**
     * @return BelongsTo<Team> Get the team for the Calendar
     */
    public function team(): BelongsTo {
        return $this->belongsTo(Team::class);
    }

    /**
     * @return HasMany Get all the relationships for the calendar.
     */
    public function shiftAssignments(): HasMany {
        return $this->hasMany(ShiftAssignment::class);
    }

    /**
     * If we want any relationship counts to be return automatically, add them here.
     */
    protected $withCount = [
        'shift_assignments'
    ];

    /**
     * Get the attributes that should be cast.
     *
     * @return array<string, string>
     */
    protected function casts(): array
    {
        return [
            'created_at' => 'datetime:Y-m-dTh:i:sZ',
            'modified_at' => 'datetime:Y-m-dTh:i:sZ',
            'deleted_at' => 'datetime:Y-m-dTh:i:sZ',
        ];
    }
}

These both produce an infinite loops:

Calendar::with('team_id', $team->id);
Calendar::all();

Laravel 8 different responses when filtering models and collections

I’m sure there is an obvious explanation here, but I’m getting weirdly different responses when filtering collections.

    $c=Media::all()->collect();
    
    // first set of commands
    $filtered=Media::all()->filter(function($item,$k){ 
      return in_array("mp4",$item->extensions->toArray();
    })->count(); // 97
    $cFiltered=$c->filter(function($item,$k){
      return in_array("mp4",$item->extensions->toArray();
    })->count(); // 96    
    Media::all()->count()==$c->count(); // true
    $ct=Media::all()->count(); // 173

    // Now - additional weirdness
    // same commands but changed filtered variable name
    
    $newFiltered=Media::all()->filter(function($item,$k){ 
      return in_array("mp4",$item->extensions->toArray();
    })->count(); // 100
    $newCFiltered=$c->filter(function($item,$k){
      return in_array("mp4",$item->extensions->toArray();
    })->count(); // 83
    Media::all()->count()==$c->count(); // true
    Media::all()->count()==$ct; // true

These were run back to back with essentially no time passing. The only difference between the first set of results and the second set is that I changed the variable names.

I’m not a Laravel developer in general and I can’t make sense of this.
Can anyone explain what’s happening here?

note: This is Laravel 8 with php-7.4. I’m executing the code in the artisan tinker shell.

How to properly use async await in my code for authentication

I have signup and login part, so when someone is signed up it should automatically forward you to login section but it does not do that even when I returned the user back to main file. I dont want to add await login() directly in signup section because i wan to keep it clean. Here is the code:

async function main() {
    while(true){
        const answer = (await question('nWelcome to the Brza Pratka shipping company. Create account if you are new one or log in to the existing one! n1. Signupn2. Loginn3. Create an order n4. ExitnChoose one: ')).trim();
        
        if(answer === '1'){ // SIGN UP SECTION
            console.log('Proceeding to Sign Up...n');
            await new Promise(resolve => setTimeout(resolve, 2000)); // wait 2 sec
            await signup();  
            const user = await login();
            await admin(user);
            break;

        } else if(answer ==='2'){ // LOGIN SECTION
            console.log('Proceeding to Log In...n');
            await new Promise(resolve => setTimeout(resolve, 2000)); // wait 2 sec
            const user = await login();
            await new Promise(resolve => setTimeout(resolve, 2000)); // wait 2 sec
            await admin(user); 
            break;

        } else if(answer === '3'){ // CREATING ORDER
            await createOrder();
            break;

        } else if(answer === '4'){ // EXIT
            rl.close();
            console.log('Goodbye');
            break;

        } else{
            console.log('❌ Invalid, you must enter from 1 to 4 to continue.n');
        }
    }
}

main();

 

Above is the main file where I connect everything via importing.

async function signup() {
const name = await askName();
const phoneNumber = await addPhoneNumber();
const email = await addEmail();
const password = await createPassword();


//////////////////////////////////
// HASHING AND SAVING THE NEW USER
const encryptedPassword = hashPassword(password); // encrypt the password

const newUser = {
    name: name, 
    phoneNumber: phoneNumber,
    email: email, 
    password: encryptedPassword
}

try {
    const NEWUSER = saveUser(newUser); // save it inside JSON
    console.log('✅ Account created. Proceeding to log in...n');
    await new Promise(resolve => setTimeout(resolve, 2000)); // wait 2 sec for login(no need but I wanted to be like in reality)
    return NEWUSER;
    
    
} catch (error) {
    console.error(error);
}

}

Above is the signup function where it takes all the required info and saving as object to the json file via saveUser function

function saveUser(user){
const users = loadFile(usersFile); // first call it to not overwrite users

users.push(user); // now add the new one
fs.writeFileSync(usersFile, JSON.stringify(users, null, 2)) // ATTENTION, if we write the file first without loading, it will overwrite thw whole file and we will lose all the users
// users, null, 2 is for better and cleaned code for reading

}

and here is the save user function to save to json

post json fetch SHOPEE AFFILIATE without API key

https://affiliate.shopee.co.id/open_api
The Shopee Affiliate Open API program has been closed
I need affiliate link generate data, and I want to do it with node js using cookies and so on but the response does not match the command on the website. is there another solution?
[enter image description here][1]

[enter image description here][2]

shopee respone
[1]: https://i.sstatic.net/TMIkB49J.png
my code respone
[2]: https://i.sstatic.net/E4Aue5EZ.png

my code `

var json_shopee={operationName:"getShopOfferLinksFromWebsite","query":"n    query getShopOfferLinksFromWebsite (n      $sourceCaller: SourceCaller!n      $shopOfferLinkParams: [ShopOfferLinkParam!]!n      $advancedLinkParams: AdvancedLinkParamsn    ) {n      shopOfferLinks (n        shopOfferLinkParams: $shopOfferLinkParamsn        sourceCaller: $sourceCallern        advancedLinkParams: $advancedLinkParamsn      ) {n        shopOfferLinkn      }n    }n  ",variables:{"shopOfferLinkParams":[{"shopId":45640084,"trace":"{"trace_id":"0.xRyfzLC89K.400","list_type":400,"root_trace_id":"0.xRyfzLC89K.400","root_list_type":400}"}],"advancedLinkParams":{"subId1":"","subId2":"","subId3":"","subId4":"","subId5":""},"sourceCaller":"WEB_SITE_CALLER"}};
fetch("https://affiliate.shopee.co.id/api/v3/gql?q=getShopOfferLinks", {
  method: 'POST',
  headers: {
   "accept": "application/json, text/plain, */*",
    "accept-language": "en-US,en;q=0.9",
    "af-ac-enc-dat": "mydata",
    "af-ac-enc-sz-token": "mydata",
    "affiliate-program-type": "1",
    "content-type": "application/json; charset=UTF-8",
    "csrf-token": "mydata",
    "priority": "u=1, i",
    "sec-ch-ua": ""Google Chrome";v="137", "Chromium";v="137", "Not/A)Brand";v="24"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": ""Windows"",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-origin",
    "x-sap-ri": "mydata",
    "x-sap-sec": "mydata",
    "Referer": "https://affiliate.shopee.co.id/offer/brand_offer",
    "Referrer-Policy": "strict-origin-when-cross-origin",
  },
  body: JSON.stringify(json_shopee),
})
  .then((response) => {
    if (!response.ok) {
      throw new Error(`HTTP error! Status: ${response.status}`);
    }
    return response.json();
  })
  .then((responseData) => {
    console.log('Success:', responseData);
  })
  .catch((error) => {
    console.error('Error:', error);
  });

`
Tell me how to post data to Shopee using cookies without an api key

Generating and downloading a binary file in iOS Safari

My web app generates binary files (byte by byte, using Uint8Array) and I want to let users save these files to their device.

I use URL.createObjectURL( Blob ), see https://jsfiddle.net/u9trmg1p

var a = document.createElement( "a" );
var blob = new Blob([  new Uint8Array([104,101,108,108,111])  ]);
var url = window.URL.createObjectURL( blob );
a.href = url;  a.download = "hello.txt";
document.body.appendChild(a);
a.click();
document.body.removeChild(a);

When you run it in the latest iOS Safari, the file appears in the Downloads section of Safari, but it can not be found on the device itself (through the Files app).

Is there a way to let a website save (“download”) files, which works in iOS Safari?

Can’t access window.open() window immediately

I have some javascript code I’m trying to run from dev tools that malfunctions. I want it to open a new tab with a particular (not cross-origin) location, to inject some javascript into that new page and allow it to run. These pieces all work separately if I paste them into the console individually, but this appears to be because of how slowly I paste them into the console. When they are put into a loop, the new window object from window.open() does not exist or is in some uninitialized state. Nor does an onload event listener work, because to the best of my understanding, if it’s uninitialized I can’t access it’s .addEventListener() method. The code in question:

let xpath = document.evaluate("//div[@class='allissues_gallerycell']/center/a[1]/@href", document, null, XPathResult.ORDERED_NODE_ITERATOR_TYPE, null);
let node;
while (node = xpath.iterateNext()) {
    let direct_link = node.value.replace(/&.+/,'&printsec=frontcover&source=gbs_ge_summary_r&cad=0');
    let tab = window.open(direct_link);
    window.addEventListener("message", function(e){
          tab.close();
    }, { once: true });
    
    tab.addEventListener("load", () => {
        let s = tab.document.createElement('script');
        s.onload = () => {
            tab.window.x();
        };
        s.src = 'https://cdn.gisthostfor.me/NoMoreNicksLeft-8EKesXvhym-gb_magazines.js';
        tab.document.body.appendChild(s);
    });
}

Running this code gives a Uncaught TypeError: Cannot read properties of null (reading 'addEventListener') and I see the dreaded red squiggles underlining tab.addEventListener(). With a human-reaction-time delay pasting each line by hand, this seems to work as intended. Is there another sort of event listener I can wrap this in so that I can set the onload? What misconception am I suffering from?

Login API takes at least 3 seconds to respond – how can I optimize it for better performance?

I’m working on a login API for travelers using Node.js (Express) and MongoDB, but it consistently takes 3 to 5 seconds to authenticate users—even with valid credentials.

I want to bring the response time below 1 second, ideally as fast as possible.

What I’ve tried:

Verified MongoDB indexes on userName, phoneNo, and email.

Used select() to limit returned fields.

Skipped validation when saving last login (validateBeforeSave: false).

Password check uses bcrypt (traveler.isPasswordCorrect()).

Tokens are generated with jsonwebtoken.

Suspected bottlenecks:

MongoDB $or query across multiple fields?

bcrypt password comparison?

Double DB calls (findOne, then findById)?

My questions:

What’s the most likely bottleneck here?

How can I profile or log the execution time of each step to pinpoint the issue?

Are there best practices to reduce login response time in a Node + Mongo + JWT stack?

Here’s my login function:

  const { userName, email, phoneNo, password } = req.body;

  if (!password) throw new ApiError(400, "Password is required");
  if (!userName && !phoneNo && !email) throw new ApiError(400, "Username or phone number is required");

  const traveler = await Traveler.findOne({
    $or: [{ userName }, { phoneNo }, { email }],
  });
  if (!traveler) throw new ApiError(404, "Traveler does not exist");

  const isPasswordValid = await traveler.isPasswordCorrect(password);
  if (!isPasswordValid) throw new ApiError(401, "Invalid Traveler Password");

  const { accessToken, refreshToken } = await generateAccessAndRefreshTokens(traveler._id);

  traveler.lastLogin = new Date();
  await traveler.save({ validateBeforeSave: false });

  const loggedInTraveler = await Traveler.findById(traveler._id).select("-password -refreshToken");

  const options = { httpOnly: true, secure: true };

  return res
    .status(200)
    .cookie("accessToken", accessToken, options)
    .cookie("refreshToken", refreshToken, options)
    .cookie("userType", "Traveler", options)
    .json(
      new ApiResponse(200, {
        traveler: loggedInTraveler,
        accessToken,
        refreshToken,
        userType: "Traveler",
      }, "Traveler Logged In Successfully")
    );
}); ```

cursors is not defined at Game.update (Phaser 3)

I’m currently learning Phaser and decided to use this Vite template (https://github.com/phaserjs/template-vite). These are the changes I’ve made:

  • Boot.js:
import { Scene } from 'phaser';

export class Boot extends Scene
{
    constructor ()
    {
        super('Boot');
    }

    preload ()
    {
        this.load.image('background', 'assets/bg.png');
    }

    create ()
    {
        this.scene.start('Preloader');
    }
}
  • Game.js:
export class Game extends Scene
{
    constructor ()
    {
        super('Game');

        this.W = window.innerWidth;
        this.H = window.innerHeight;
    }
    create () {
        let player = this.add.rectangle(this.W / 2, this.H / 2, 25, 25, 0xfafafa);
        this.physics.add.existing(player);
        const cursors = this.input.keyboard.createCursorKeys();
        this.cameras.main.setBackgroundColor(0x0a0a0a);
    }

    update () {
        
        if (cursors.left.isDown) { // Error happens here
            player.body.setVelocityX(-20);
        }

    }
}

However, when I run it, this error occurs: Uncaught ReferenceError: cursors is not defined at Game.update (Game.js:26:9)

I’ve tried to make the player variable and the cursors variable global and in different places in the game.js file, but it didn’t work. The error didn’t change at all.

bypass the os download location while downloading pdf using @react-pdf/renderer

I have used @react-pdf/renderer and while downloading pdf it gives os download choose location popup i want to bypass it or show toast message in my system when download is complete.
I have tried this

const handleDownload = async () => {
  try {
    const blob = await pdf(
      <MyDocument data={data} startDate={startDate} endDate={endDate} currency={currency} />
    ).toBlob();

    const url = URL.createObjectURL(blob);
    const link = document.createElement("a");
    link.href = url;
    link.download = `DetailsSalesReport_${moment().format("YYYY-MM-DD_HHmmss")}.pdf`;
    document.body.appendChild(link);

    link.click();
    toast.success("Download started! Check your Downloads folder.");

    link.remove();
    URL.revokeObjectURL(url);
  } catch (error) {
    toast.error("Failed to download PDF");
    console.error("PDF download error:", error);
  }
};
<PDFDownloadLink
        document={<MyDocument data={data} startDate={startDate} endDate={endDate} currency={currency} />}
        fileName={`DetailsSalesReport_${moment().format("LLLL")}.pdf`}
      >
        {({ loading }) => (
          <div className="p-1 border mt-1 flex justify-center items-center text-center" onClick={handleDownload}>
            <p>{loading ? "Loading document..." : "Download"}</p>
            &nbsp; &nbsp;
            <img src={DownloadIcon} alt="Download" />
          </div>
        )}
      </PDFDownloadLink>

But it shows toast message when the popup is triggered. How can we get toast message when download is complete totally

Angular HttpClient interceptor: Error 429 returns status 0 instead of 429

I am working with an Angular HTTP interceptor and handling various error status codes. However, I have an issue with handling the 429 (Too Many Requests) error code. While the backend correctly responds with a 429 status in tools like Postman, in Angular, I receive errorResponse.status = 0 instead of 429 when the request returns this error.

When the backend responds with status codes 200, 500, or 401, my interceptor handles the errors correctly. But when the server responds with 429, errorResponse.status is set to 0.

I tested the same request in Postman, and the server correctly returns a 429, but this is not happening in Angular.

Here is a snippet of the interceptor code:

   intercept(request: HttpRequest<any>, next: HttpHandler): Observable<HttpEvent<any>> {
        let accessToken = sessionStorage.getItem("access_token");
        if (request.url !== './assets/config/env.json') {
            request = request.clone({
                setHeaders: {
                    Authorization: `Bearer ${accessToken}`,
                    sid: `${this.config.getEnv('sid')}`,
                }
            });
        }
        return next.handle(request).pipe(
            tap(
                event => this.handleResponse(request, event),
                error => this.handleError(request, error)
            )
        );
    }
    handleError(req: HttpRequest<any>, errorResponse: HttpErrorResponse) {
        if (errorResponse.status === 429) {
            this.mostrarMensajeNoDuplicado("You have reached the user request limit, try again later.");
        } 
    }

How can I register a new event listener with a function without it overriding the old listener?

So I’ve been trying to make a game to learn some javascript, but whenever I try to create a new event listener through a function, e.g.

function event(target) {
    target.addEventListener('blur', doStuff);
}

then changing the target argument simply overrides any other event listeners created with the same function.

I have tried putting the event listener into its own function and tried adding arguments, as well as messing around with ordering, but nothing I do seems to work.

If you’ve encountered a situation where: Fetch works, response JSON is correct, but Nothing renders into the DOM [closed]

I’m running into a persistent issue where my frontend (plain HTML + JS) is sending a request to a FastAPI backend, receiving a valid JSON response, but nothing is rendered into the DOM.

Despite trying many debugging approaches, the final response just won’t appear in the designated <div>. Here’s a simplified breakdown.


✅ What works

  • HTML interface loads cleanly with a text input, submit button, and an empty <div id="replyPanel">.
  • JS captures the input and calls:
    fetch('http://127.0.0.1:8000/ask', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ prompt: userInput })
    })