Python – scraping files that are in JavaScript objects

I’m trying to download files from a VA dataset website with a Python scraper but I’m having trouble figuring out how to parse the JavaScript in the HTML website that appears to contain the files. This is source code for the website (view-source:https://www.data.va.gov/dataset/Air-Force-Veterans-2017-Living-Only/9u8y-zaby). I’m trying to download the “.xlsx” files, which (by just using command+F on my Mac) I think are in JavaScript objects. I’ve looked around this site and others but haven’t been able to figure out how to scrape links from within JavaScript. How should I go about doing this? Any help would be greatly appreciated.

MUI DataGridPro crashes when reordering columns by drag&drop, when already using react-dnd in the parent component

My app is a dashboard of MUI <Card />s that can be dragged-and-dropped (d&d) to reordem them. The d&d logic is implemented using react-dnd and has been working well so far.
However, when I add a <DataGridPro /> as the child of a draggable <Card />, the datagrid’s native Column ordering – which also is done by dragging-and-dropping – breaks. Dragging a column once or twice generates the following crash:

Invariant Violation
Cannot call hover while not dragging.
▼ 5 stack frames were expanded.
    at invariant (https://bfz133.csb.app/node_modules/
react-dnd/invariant/dist/index.js:19:15
checkInvariants
https://bfz133.csb.app/node_modules/dnd-core/dist/actions/dragDrop/hover.js:33:40
DragDropManagerImpl.hover
https://bfz133.csb.app/node_modules/dnd-core/dist/actions/dragDrop/hover.js:18:5
Object.eval [as hover]
https://bfz133.csb.app/node_modules/dnd-core/dist/classes/DragDropManagerImpl.js:25:38
HTML5BackendImpl.handleTopDrop
https://bfz133.csb.app/node_modules/react-dnd-html5-backend/dist/HTML5BackendImpl.js:455:20
▲ 5 stack frames were expanded.
This screen is visible only in development. It will not appear if the app crashes in production.
Open your browser’s developer console to further inspect this error.
This error overlay is powered by `react-error-overlay` used in `create-react-app`.

You can begin dragging, the crash only happens when you let go of the mouse button.

The expected behavior is that I should be able to d&d the columns, to change their order, without issues.

Things I’ve tried

  • Removing the <DataGridPro /> and replacing that <Card /> with a text-type Card (see the code in the sandbox below) shows that d&d logic works fine with no crashes;
  • Disabling my app’s d&d by commenting out all the relevant code causes the <DataGridPro />‘s colum reordering to work as expected;

The above suggests the root cause lies in having both D&Ds work without causing an internal conflict in react-dnd, which let to me trying:

  • Browsing the documentation to find a way to instruct the component to use my own DndProvider or DndManager, but I couldn’t find that in the API – sorry if I misread it!
  • Googling for the error message “Cannot call hover while not dragging”, while limiting myself to contexts including the MUI library or react-dnd, yielded limited results. I found a Chrome bug that was fixed on v. 77.0.3865.120, my Chrome version is 101.0.4951.64 .

Minimum verified reproducible example

I made a sandbox! Click here to see it

How to display localStorage value in a table header?

I have the following js function that is used to get the saved value of an input field:

function getSavedValue(e) {
    if (!localStorage.getItem(e)) {
        return "";
    }
    return localStorage.getItem(e);
}

The following is an example of how I use this js function:

var paymentMonth = document.getElementById("monthHR_PaymentMonth");

paymentMonth.value = getSavedValue("monthHR_PaymentMonth");

How can I use this function to grab the paymentMonth value and display it in a table column header?:

<th>Payment May 2022</th>

Where May 2022 is the value saved.

ScrollView not scrolling inside absolute component

I’m trying to create a fixed height scrollview from a select button, in with will be able to be scrolled. Although I can’t seem to make the scroll work inside a View component.
I have the following component:

<MyInput
        key={name}
        type={'select'}
        rightIcon={rightIcon}
        placeholder={"Some"}
        onChangeText={onChangedText}
        onInputFocus={onShowDropdown}
        dropdown={active && options.length > 0 && (
          <styled.Dropdown>
            <styled.Options
              keyboardShouldPersistTaps='handled'
              scrollEnabled={(options.length > 3)}
              showsHorizontalScrollIndicator={false}
              showsVerticalScrollIndicator={false}
              horizontal={false}>
              {options.map((item) => (
                <Option
                  key={item.id || item.text}
                  title={item.place_name || item.text}
                  selected={(item.id === highlighted) || (item.text === highlighted)}
                  onHideUnderlay={() => onHideUnderlay(item.id || item.text)}
                  onShowUnderlay={() => onShowUnderlay(item.id || item.text)}
                  onPressed={() => onSelectOption(item)} />
              ))}
            </styled.Options>
          </styled.Dropdown>

When I remove the </styled.Dropdown> the List work fine, but it won’t have a fixed height.

This is the styles:

styles.Dropdown = styled.View`
  position: absolute;
  left: 0;
  right: 0;
  top: 51px;
  z-index: 30;
  background-color: ${Colors.white};
  box-shadow: 0 4px 8px rgba(0, 0, 0, 0.12);
  min-height: 56px;
  max-height: 168px;
  border-bottom-left-radius: 8px;
  border-bottom-right-radius: 8px;
`

styles.Options = styled.ScrollView`
  flex: 1;
  border-bottom-left-radius: 8px;
  border-bottom-right-radius: 8px;
`

styles.Option = styled.TouchableHighlight``

This is how it renderes (with the complete code), but it doesn’t allow scrolling and the selection doesn’t work:
enter image description here

This is how it renderes without the <styled.DropDown> and it works, but without a fixed height:
enter image description here

Map function not running after useEffect sets the state to a non-empty list

I am trying to fetch some data from my node js server that returns to me an array. Upon receiving this array, I want to map through it and display several iterations of the same component with this data. However, the map function does not run as the initial state value I set was an empty array. My question is, why isn’t the map function running after the state is changed in the useEffect function? Here is the code:

const [groupData, setGroupData] = useState([]);

useEffect(() => {
    const func = async () => {
        const fetchData = await fetch("http://localhost:5000/api/get-groups", {
            method: "GET",
            credentials: "include",
            headers: {
                "Content-Type": 'application/json',
                'Access-Control-Allow-Methods': 'GET,PUT,POST,DELETE,PATCH,OPTIONS'
            }
        })

        const jsonFetch = await fetchData.json();
        setGroupData(jsonFetch.groupData);
    }

    func();
}, [])

return (
    {groupData.map((elem) => {
        <Card data={elem} />
    })}
)

But the cards are simply not visible. I can console.log to see the data coming into the react act as expected. I also tried changing the second parameter of the useEffect function to [groupData] instead of []. But it still does not work. Any help would be appreciated. Thank you!

i don’t know how to add the user input back to the DOM and i think its something with my js code

i don’t konw what to do
i need help with this js script and i just need to know how to show a user input in a html styled templet that i made but i just don’t don’t known how

<!-- add email side nav -->
<div id="side-form" class="sidenav side-form">
  <form class="add-email container section">
    <h6>add a new email</h6>
    <div class="divider"></div>
    <div class="input-field">
      <input keypath="email" placeholder="[email protected]" id="title" type="text" class="validate">
      <label for="title">write email here</label>
    </div>
    <div class="input-field">
      <label>
        <input type="checkbox" />
        <span>Notifications</span>
      </label>
    </div>
    <br>
    <div class="input-field center">
      <button class="btn-floating blue pulse" id="add_email" onclick="getValue()">Add</button>
    </div>
  </form>
</div>

How do I display a popup modal after submission?

I’m following a tutorial on modals and components. I have a form that contains a submit button at the end that when clicked, should display a small modal with a thank you for submitting!

<form onSubmit={onSubmit}>
  <div className='form-group'>
     <label htmlFor='resource'>Resource</label>
       <input className='form-control' id='resource'
        placeholder='resource in question' required />
            </div>
               <div className='form-group'>
                 <label htmlFor='description'>Description</label>
                <textarea className='form-control' id='description' required />
            </div>
            <div className="form-group">
       <button className="form-control btn btn-primary" type="submit" >
          Submit
       </button>
   </div>
</form>

Here is the other files associated.
The form component is imported into my Modal.js

import React from 'react';
import '../modal.css';
import ReactDOM from 'react-dom';
import { Form } from './reportComponent';
import FocusTrap from 'focus-trap-react';
export const Modal = ({
  onClickOutside,
  onKeyDown,
  modalRef,
  buttonRef,
  closeModal,
  onSubmit
}) => {
  return ReactDOM.createPortal(
    <FocusTrap>
      <aside
        tag="aside"
        role="dialog"
        tabIndex="-1"
        aria-modal="true"
        className="modal-cover"
        onClick={onClickOutside}
        onKeyDown={onKeyDown}
      >
        <div className="modal-area" ref={modalRef}>
          <button
            ref={buttonRef}
            aria-label="Close Modal"
            aria-labelledby="close-modal"
            className="_modal-close"
            onClick={closeModal}
          >
            <span id="close-modal" className="_hide-visual">
              Close
            </span>
            <svg className="_modal-close-icon" viewBox="0 0 40 40">
              <path d="M 10,10 L 30,30 M 30,10 L 10,30" />
            </svg>
          </button>
          <div className="modal-body">
            <Form onSubmit={onSubmit} />
          </div>
        </div>
      </aside>
    </FocusTrap>,
    document.body
  );
};

export default Modal;

The Modal is imported into the container:

import React, { Component } from 'react';
import { Modal } from './Modal';
import TriggerButton from './triggerbutton';

export class Container extends Component {
  state = { isShown: false };
  showModal = () => {
    this.setState({ isShown: true }, () => {
      this.closeButton.focus();
    });
    this.toggleScrollLock();
  };
  closeModal = () => {
    this.setState({ isShown: false });
    this.TriggerButton.focus();
    this.toggleScrollLock();
  };
  onKeyDown = (event) => {
    if (event.keyCode === 27) {
      this.closeModal();
    }
  };
  onClickOutside = (event) => {
    if (this.modal && this.modal.contains(event.target)) return;
    this.closeModal();
  };

  toggleScrollLock = () => {
    document.querySelector('html').classList.toggle('scroll-lock');
  };
  render() {
    return (
      <React.Fragment>
        <TriggerButton
          showModal={this.showModal}
          buttonRef={(n) => (this.TriggerButton = n)}
          triggerText={this.props.triggerText}
        />
        {this.state.isShown ? (
          <Modal
            onSubmit={this.props.onSubmit}
            modalRef={(n) => (this.modal = n)}
            buttonRef={(n) => (this.closeButton = n)}
            closeModal={this.closeModal}
            onKeyDown={this.onKeyDown}
            onClickOutside={this.onClickOutside}
          />
        ) : null}
      </React.Fragment>
    );
  }
}

export default Container;

Would it be best to just make a new modal and import and call that when the submit button is clicked or is there a better way?

Hosted build react app doesn’t work while it does on local

I’ve been trying to develop a React app.
I perfectly works on my localhost:3000 but if I try to start the build version – either by hosting it on Hostinger or with ‘serve -s build’ command – it shows an empty black page.
The console has the following erros:

main.fc1df3be.js:2 Error: Invariant failed
at main.fc1df3be.js:2:723458
at Lg (main.fc1df3be.js:2:723472)
at main.fc1df3be.js:2:724672
at Vg (main.fc1df3be.js:2:724679)
at oo (main.fc1df3be.js:2:173757)
at Qs (main.fc1df3be.js:2:226057)
at Cu (main.fc1df3be.js:2:213298)
at Tu (main.fc1df3be.js:2:213226)
at Nu (main.fc1df3be.js:2:213089)
at bu (main.fc1df3be.js:2:210055)
cs @ main.fc1df3be.js:2
n.callback @ main.fc1df3be.js:2
pa @ main.fc1df3be.js:2
gs @ main.fc1df3be.js:2
Ru @ main.fc1df3be.js:2
t.unstable_runWithPriority @ main.fc1df3be.js:2
Ki @ main.fc1df3be.js:2
Pu @ main.fc1df3be.js:2
bu @ main.fc1df3be.js:2
pu @ main.fc1df3be.js:2
el @ main.fc1df3be.js:2
(anonymous) @ main.fc1df3be.js:2
ku @ main.fc1df3be.js:2
ol @ main.fc1df3be.js:2
t.render @ main.fc1df3be.js:2
(anonymous) @ main.fc1df3be.js:2
(anonymous) @ main.fc1df3be.js:2
(anonymous) @ main.fc1df3be.js:2
main.fc1df3be.js:2 

Uncaught Error: Invariant failed
at main.fc1df3be.js:2:723458
at Lg (main.fc1df3be.js:2:723472)
at main.fc1df3be.js:2:724672
at Vg (main.fc1df3be.js:2:724679)
at oo (main.fc1df3be.js:2:173757)
at Qs (main.fc1df3be.js:2:226057)
at Cu (main.fc1df3be.js:2:213298)
at Tu (main.fc1df3be.js:2:213226)
at Nu (main.fc1df3be.js:2:213089)
at bu (main.fc1df3be.js:2:210055)

I have no idea as I am a beginner.
I have a lot of packages installed – including @metamask, web3 & bunch of others. I saw online that it can be linked to a bad installation of these packages.

I’d be more than happy to hear from you guys & provide more infos if needed 🙂

Best,
z

How to wait a promise before returning a value in a function javascript?

In my code below, I want to resolve the country location of a user, using google geocode API.
The country variable is set after the axios function is called.
However, getCountry() returns an empty object. How do return the value only after country has been updated?

Trying to get a hang of asynchronous functions in Javascript.
Any help is appreciated,thanks.

import axios from "axios";

export const getCountry = () => {
  const pos = {};
  let country = {};

  if (navigator.geolocation) {
    navigator.geolocation.getCurrentPosition(findLocal, showError);
  } else {
    console.log("Geolocation is not supported by this browser");
  }

  async function findLocal(position) {
    pos.lat = position.coords.latitude;
    pos.lng = position.coords.longitude;

    await axios(`https://maps.googleapis.com/maps/api/geocode/json?latlng=${pos.lat},${pos.lng}&key=${process.env.REACT_APP_GOOGLE_API_KEY}
      `).then((data) => {
      if (data?.data?.status === "OK") {
        let address_components = data.data.results[0].address_components;
        for (var i = 0; i < address_components.length; i++) {
          if (address_components[i].types[0] === "country") {
            country = {
              long_name: address_components[i].long_name,
              short_name: address_components[i].short_name,
            };
            break;
          }
        }
      }
    });
  }

  function showError(Error) {
    console.log(Error);
  }

  return country;
};


console.log(getCountry()); // actual_result = {} intended_result = {long_name: value, short_name: value}

Is using react hooks the proper way to dynamically add menu items from parsed json?

I am new to React/JavaScript but feel like I am learning decently, so bear with me.

I have a React web app site that loads a JSON file and displays the data into groupings like so.
enter image description here
If it matches a specific file name, it labels it “Full Configuration”, else it just displays the file name on the menu item.

After the JSON file is parsed, a div with unique ID is displayed for each grouping. I keep an array of all the div IDs.

What I would like to do, is append a menu item in the menu to view a single div of the “Full Configuration” file. Lets say the user only wants to see the GPS Lever Arms, there would be a menu item for “GPS Lever Arms” they could click to change the display property of all divs to “none” except for that one. So the user would then see this.
enter image description here

I have done some looking the past few days and it seems like the useEffect is the correct way to go about this but I have not seen it implemented in my desired way.

This is my current code that receives a json object data and creates the div per group and then displays each item inside the group.

import React, {useState} from 'react'
import ConfigParam from './ConfigParam'

export default function Grouping({data}){
    const [jsonData, setJsonData] = useState(data)

    // Array of arrays To hold the grouping results.
    var groupResults = {};
    var divIDs = [];

    for(var obj in data)
    {   
        if(data[obj].hasOwnProperty('meta') === false)
        {
            console.log(`no meta field in ${data[obj]}`);
            return <></>
        }
        else if(data[obj].meta === "")
        {
            console.log(`${data[obj]} meta field is empty`);
            return <></>
        }
        else
        {
            // check if the group property is missing - create it as empty
            if(data[obj].meta.hasOwnProperty('group') == false)
            {
                data[obj].meta.group = ""
            }

            if(!groupResults.hasOwnProperty(data[obj].meta.group))
            {
                groupResults[data[obj].meta.group] = [];
            }
            // take the name of the key and add it as a property inside the object
            data[obj].name = obj;
            // push the object onto the json
            groupResults[data[obj].meta.group].push(data[obj]);
        }
    }

    const editValue = ({name,data}) =>
    {
        var temp = jsonData;
        temp[name].value = data;
        //console.log(data)
        setJsonData(temp)
    }

    // Get the keys of the groupResults (Group names)
    const test = Object.keys(groupResults);

    return(
        test.map((member,index) =>{
            // get the keys within the current group (items)
            const subItems = Object.keys(groupResults[member]);

            // create a div id per group
            let div_id = "group-" + test[index];
            divIDs.push(div_id);
            console.log("new divID:", div_id);

            // todo - append button for each div to view each individual div. 

            return(
                <div id={div_id} style={{display:"flex", border: "1px black solid", marginBottom: "10px"}}>
                    <div style={{flex:4, margin: "10px"}}>
                        <div style={{margin: "10px"}}>
                            <h4><b>{test[index]}</b></h4>
                        </div>
                        <div style={{margin: "30px"}}>
                            {subItems.map((sub, index2)=>{
                                return(
                                    <ConfigParam
                                        editValue={editValue}
                                        key={index2}
                                        name={groupResults[member][sub].name}
                                        type={groupResults[member][sub].meta.type}
                                        value={groupResults[member][sub].value}
                                        description={groupResults[member][sub].meta.description}
                                        meta={groupResults[member][sub].meta} />
                                )
                            })}
                        </div>
                    </div>
                </div>
            )
        })
    )
}

for context, the JSON looks like this:

{
  "GPS Lever Arm X (ft)": {
    "value": 1.78,
    "meta": {
      "group": "GPS Lever Arms",
      "description": "GPS <--> IMU",
      "type": "number",
      "readOnly": false,
      "min": -1.7976931348623157e+308,
      "max": 1.7976931348623157e+308
    }
  },
  "GPS Lever Arm Y (ft)": {
    "value": 0,
    "meta": {
      "group": "GPS Lever Arms",
      "description": "GPS <--> IMU",
      "type": "number",
      "readOnly": false,
      "min": -1.7976931348623157e+308,
      "max": 1.7976931348623157e+308
    }
  },
  "GPS Lever Arm Z (ft)": {
    "value": 0,
    "meta": {
      "group": "GPS Lever Arms",
      "description": "GPS <--> IMU",
      "type": "number",
      "readOnly": false,
      "min": -1.7976931348623157e+308,
      "max": 1.7976931348623157e+308
    }
  }
}

In the event someone would like to see the menu code, here –
Side note: the “grouping” file is called from the ConfigFile. – I can provide that code if needed.

import React from 'react'
import { Tab, Row, Col, Nav} from 'react-bootstrap'
import FunctionPanel from './FunctionPanel'
import VariablePanel from './VariablePanel'
import ConfigFile from './ConfigFile'

export default function MainViewColumns({groups}) {

    const member = groups[1];
    const files = member.files;
    const funcs = member.functions;
    const vars = member.variables;

    var defaultKey="controls"
    var menuItems = [];
    var viewItems = [];

    if(files)
    {   
        for(var x = 0; x < files.length; x++)
        {
            // Get the file name info
            const fullFileName = files[x].split('/').pop();
            const fileInfo = fullFileName.split(".");
            const fileExtension = fullFileName.split(".").pop();
            const filename = fileInfo[0];

            // create the menu item based on filename
            if(fullFileName === "atacnavMicroConfig.json")
            {
                defaultKey=filename;
    
                var menuItem = 
                    <Nav.Item key={filename}>
                        <Nav.Link eventKey={filename}>Full Configuration</Nav.Link>
                    </Nav.Item>
            }
            else
            {
                menuItem = 
                    <Nav.Item key={filename}>
                        <Nav.Link eventKey={filename}>{filename}</Nav.Link>
                    </Nav.Item>
            }

            // create the view item
            var viewItem = 
                <Tab.Pane key={filename} eventKey={filename} >
                    <ConfigFile file={files[x]}/>
                </Tab.Pane>

            // append menu item and view item to their respective lists. 
            menuItems.push(menuItem);
            viewItems.push(viewItem);
        }
    }

    return (
        <Tab.Container defaultActiveKey={member.name}>
            <Tab.Container defaultActiveKey={defaultKey}>
            <Row>
                <Col sm={2}>
                    <div id="navColumn">
                        <Nav variant="pills" className="flex-column">
                            <Nav.Item id="mainMenuItem">Main Menu</Nav.Item>
                            <Nav.Item key="controls">
                                <Nav.Link eventKey="controls">Controls</Nav.Link>
                            </Nav.Item>
                            {menuItems.map((item, index) => {
                                return(menuItems[index]);
                            })}
                        </Nav>
                    </div>
                </Col>
                <Col sm={9}>
                    <Tab.Content>
                        <Tab.Pane key="controls" eventKey="controls" >
                            <div style={{display: "flex", justifyContent:"space-between"}}>
                                <div style={{flex: 8, marginTop: "10px"}} className="d-flex flex-col justify-content-center">
                                    <FunctionPanel functions={funcs}/>
                                    <VariablePanel variables={vars}/>
                                </div>
                            </div>  
                        </Tab.Pane>
                        {viewItems.map((item, index) => {
                            return(viewItems[index]);
                        })}
                    </Tab.Content>
                </Col>
            </Row>
        </Tab.Container>
        </Tab.Container>
    )
}

Is useEffect the proper way to accomplish this ? if so, how would I go about implementing it?
Thanks in advance.

Check if JSON Object starts with ‘https’ in JavaScript

I have urls like this:

"link": "https://en.wikipedia.org/wiki/Antigua_and_Barbuda"
"url": "https://en.wikipedia.org/"

I want to write an if statement that checks if the object starts with a ‘https:’ and then tell it what to do.

Note: jsonObject in my code represents the objects in the returned json data. This is what I have, but it doesn’t make my urls clickable or returns its data.

if (jsonObject.startsWith("http://")) {
    const text = document.createElement("a");
    k.textContent = jsonObject;
    k.setAttribute("href", jsonObject);
    j.appendChild(k);

How do I improve the rendering speed of a component that renders an svg with a polygon with points based on user input?

I have a component that I call AnswerBetaComponent. It returns a beta distribution (a mathematical function, a probability distribution, which could just as well be any other function), based on the inputs of the user. A beta distribution is dependent upon two variables, which the user decides by interacting with the screen, alpha and beta.

By swiping right, the user increases alpha and decreases beta. By swiping left the user does the opposite. By swiping upwards the user increases alpha and beta proportionately.

When alpha and beta are set, I calculate a new set of points based on those values in my AnswerBetaComponent. I then return another component called BetaGraph, responsible for displaying the function in the form of an svg.

A general explanation of the AnswerBetaComponent can be found below.

export const AnswerBetaComponent = (props) => {
...

// calculating alpha and beta through onMouseMove and onTouchMove event handlers.
...

  return (
    <div className="answer-component-container">
      <div
        onMouseMove={mouseMove}
      >
        <BetaGraph>
          <polygon
            points={pointsDeterminedByUserInteraction}
            onTouchMove={swipeMove}
          />
        </BetaGraph>
      </div>
    </div>
  );
};

The BetaGraph component looks somewhat like this:

const BetaGraph = (props) => {
  return <svg>{props.children}</svg>
}

Since the polygon with the points defined in AnswerBetaComponent is passed as a child to BetaGraph, it gets displayed through the svg.

The problem I have is that the function renders quite slow, especially on mobile devices, meaning that it feels somewhat laggy when a user swipes the function left, right, up or down.

Is there any way I can increase the rendering speed of my component, such that what is displayed on the screen is a beta distribution which can be seamlessly drawn to any side of the screen?

The function runs once, the loop inside it runs 2 times?

I simplified the code to explain the problem. When I click the button in my program 3-4 times and the last time I click it, messageStatus = true, the loop runs 3-4 times.

ipcMain.on('sendMessageMail', async (err, data) => {
    if (messageStatus) {
        console.log('Stop');
        messageStatus = false;
        return;
    }

    if (!messageStatus) {
        console.log('Start');
        messageStatus = true;
    }

    for (let i = 0; messageStatus && i < 100; i++) {
        console.log(i);
        await sleep(1000);
    }
})

Console Output

What is the problem? Why is the loop running 3-4 times while the event is running once?