Use Python to crawl Zhihu's god reply about programmers

Use Python to crawl Zhihu's god reply about programmers

Data Science Club

Chinese data scientist community

Author : Brother Qiang, once worked for Morgan Stanley and eBay.

\

Crawling Zhihu's reply is very simple. In this article, we will reveal the principle behind it.

Know what are the characteristics of God's reply? Let's take a look


\

Do you see any pattern? Is it short and incisive? Do you agree with a lot? Therefore, crawling Zhihu God replies to us as long as we crawl the answers that are more agreeable and have a few words. It can be achieved in two simple steps. The first step is to crawl Zhihu answers, and the second is to screen answers. Isn't it easy?

\

Crawl to know the answer

The first step is to crawl Zhihu's answer. There are too many answers on Zhihu, and crawling all the answers at once will be time-consuming. We can select a few topics and crawl the content of these topics.
The following function is used to crawl the content of a specific topic

def get_answers_by_page ( topic_id, page_no ):    offset =    page_no* 10 url = <topic_url> # topic_url is the url corresponding to this topic    headers = {        "User-Agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" ,    }    r = requests.get(url, verify = False , headers=headers)    content = r.content.decode( "utf-8" )    data = json.loads(content)    is_end = data[ "paging" ][ "is_end" ]    items = data[ "data" ]    client = pymongo.MongoClient()    db = client[ "zhihu" ]     if len (items)> 0 :        db.answers.insert_many(items)        db.saved_topics.insert ({ "topic_id" : topic_id, "page_no" : page_no})     return is_end duplicated code

The get_answers_by_page function has two parameters. The first parameter is the id of the topic, and the second parameter indicates the content of the page to be crawled.
There are several fields that need attention in the crawled content. The yellow boxes are highlighted in the figure below.

The meaning of these fields is as follows:

  • question.title-the title of the question
  • content-the content of the answer
  • voteup_count-the number of votes

These fields will be used in the next step of filtering answers.

\

Filter answers/

After crawling the data, let's filter the results.
We use the aggregation pipeline in MongoDB to filter the answers (for the usage of the aggregation pipeline in MongoDB, please refer to the Aggregation Pipeline Quick Reference article at https://docs.mongodb.com/manual/meta/aggregation-quick-reference/),code show as below

client = pymongo.MongoClient() db = client[ "zhihu" ] items = db.answers.aggregate([    { "$match" : { "target.type" : "answer" }},    { "$match" : { "target.voteup_count" : { "$gte" : 1000 }}},    { "$addFields" : { "answer_len" : { "$strLenCP" : "$target.content" }}},    { "$ Match" : { "answer_len" : { "$ LTE" : 50 }}},]) copy the code

The above code will filter all the answers that agree with greater than 1000 and the number of words is less than 50, and the result of the filtering is a short and brilliant answer.

The above is part of the core code. Due to the limited space, you can reply to " Zhihu " in the background of the official account data science club to obtain the complete code download address.

Knowing God Reply

After the code is written, let's run it. It happened to be the Programmer's Day yesterday, so let's filter out the replies related to programmers. The result is as follows, a total of 75 funny paragraphs?
\

1

\

Q: What are the "lie" that programmers say most often?

\

A://TODO

\

2

\

Q: What is the experience of staying green on GitHub for 365 days?

\

A:

I used to stay green for more than 200 days, but I left my girlfriend in the cold and stayed green until now.

\

\

3

\

Q: How to refute the view that "programmers leave the computer is waste"?

\

A: No, no, no, many programmers are rubbish in front of the computer.

\

\

4

\

Q: If one day all people spoke in computer language, what would it be like?

\

A:

hello, world.

\

\

5

\

Q: Suddenly I want to open a programmer-themed restaurant. The name is the programmer's dish, and the name of the dish is the keyword in various languages. Please give me some suggestions. Is there a future?

\

A: When you enter the door, a big hello world 
signature dish called "Braised Product Manager"
will definitely be full

\

\

6

\

Q: What is recursion?

\

A: The definition and scope of "political content not suitable for public discussion" also belong to "political content not suitable for public discussion"

\

\

7

\

Q: How to translate the most basic term "bug" in programming?

\

A:

Yaomozi, your program has another Yaomozi.

\

\

8

\

Q: What is the joy of programming?

\

A: People's sense of accomplishment comes from two things, creation and destruction.

\

\

9

\

Q: How to refute the view that "programmers leave the computer is waste"?

\

A: Honestly, you can talk to this kind of woman, do you want to be on her?

\

\

10

\

Q: As a programmer, what mathematical losses did you suffer when programming?

\

A: I was "obviously" pushed for an afternoon when I was reading the paper

\

\

11

\

Q: What equipment do local programmers have?

\

A: Girlfriend. . .

\

\

12

\

Q: Which god should I worship if there is no bug in the code?

\

A: Adore Yongzheng and treat Ba Ge.

\

\

13

\

Q: Is it the only way for children from poor families in China to be promoted to the middle class by being admitted to a good university and studying IT?

\

A: Yes, just 4 ways to
write code,
engage in finance, engage in finance
in code
circle, write code in financial circle

\

\

14

\

Q: Why do programmers like to carry a computer bag wherever they go, even if there is no computer in it?

\

A: Because they have no other bags.

\

\

15

\

Q: How to translate "Talk is cheap. Show me the code" better?

\

A: Don't talk nonsense, just come here.

\

\

16

\

Q: Why are programmers girlfriends or wives generally more attractive than their male counterparts? Or is it that programmers are already considered high-quality stocks in the marriage and love market?

\

A: The programmer s girlfriend has a good look. I m the one who takes it, because if you ask ten programmers who their girlfriend is, nine of them answered Yui Aragaki.

\

\

17

\

Q: Why do some people prefer to buy a few mechanical keyboards for use instead of putting a mask on themselves?

\

A:

Lao Tzu does not rely on his face to eat.

Lao Tzu s hard-earned banknotes. I spend whatever I want.

\

\

18

\

Q: What kind of words should be engraved for the programmer's couple wedding ring?

\

A: 0 error 0 warning

\

\

19

\

Q: Will IT engineers feel uncomfortable when they are called "code farmers"?

\

A: We are still human beings anyhow, products and designs are already dogs...

\

\

20

\

Q: Why would a sales man (30 years old) ask me a male programmer (24 years old) to go to Starbucks near the community?

\

A: Based on his many years of experience, he should have a huge idea and only a programmer has to implement it.

\

\

21

\

Q: How do I find a girl who likes programmers to be girlfriends?

\

A: Look at Fate, I know that there are so many users, you pay attention to me, it is Fate.

\

\

22

\

Q: How does the programmer girlfriend celebrate the programmer boyfriend's birthday?

\

A: Tell him that the interface is ready.

\

\

23

\

Q: As a programmer, how did you find a girlfriend after work?

\

A: The subject has been a programmer for so long, and it is commendable to still like girls.

\

\

24

\

Q: What preparations do programmers need to make when they switch to barbecue? What are the advantages and disadvantages?

\

A: You see, you don t even know where your advantages and disadvantages are, so you still need a product manager.

\

\

25

\

Q: What can irritate programmers?

\

A: I said when I passed by his computer, Yo, I was writing a bug again!

\

\

26

\

Q: One of my teachers said that Java is suitable for large-scale software and C# is suitable for small and medium-sized software. Is this true?

\

A: Java has a talent, that is, it can write small and medium-sized software into large-scale ones.

\

\

27

\

Q: Why is the salary of programmers so high in 2014?

\

A: The hourly salary is not high

\

\

28

\

Q: Are most programmers complaining about low wages?

\

A:

Who, who is complaining about high wages?

\

\

29

\

Q: What should I do if the single programmer solves a technical problem and no girl can show off or praise himself?

\

A: Now you understand why so many programmers write technical blogs.

\

\

30

\

Q: Do Chinese programmers prefer "Jackets+Jeans+Sports Shoes" clothing? If so, why is there such a trend?

\

A: Does it look so good for the program?

\

\

31

\

Q: As an IT practitioner, what tools do you think have greatly improved your work efficiency?

\

A:

single

\

\

32

\

Q: Why do I think programmers seem to be mostly bad words?

\

A:

Just take it as our EQ is low,

So you are happy,

We are also happy.

\

\

33

\

Q: In China, the oldest programmer is only about 40 years old. What else can Chinese programmers do in the future?

\

A:

This is the same principle as why no one after the 90s has lived beyond 30.

\

\

34

\

Q: How to reply to the text message sent by the programmer: "Hello world"?

\

A: hello nerd.

\

\

35

\

Q: How do you know that an IT guy likes a girl?

\

A: When he fights the habit of reticence that he has cultivated a long time ago to get close to you desperately

\

\

36

\

Q: Why shouldn't programmers repair computers?

\

A: Does Fan Bingbing need to be able to repair TVs?

\

\

37

\

Q: My colleague said that his C++ level is the first in China. How did he realize that he is not so good?

\

A:

To be honest, I am not pretending to be forced: my C++ level is 0 in the country.

\

\

38

\

Q: Why do all the icons shake when deleting software on iPhone?

\

A: The third-party software is scared, and the software that comes with the system is awkward

\

\

39

\

Q: The revolver contains a bullet. Shooting one shot at your head will reward you with a reward of 100,000 yuan, two shots 100 million, three shots 200 million, four shots 400 million, and 5 shots 1.6 billion. Is it worth it?

\

A:

As long as I don t hit the key, let me tell you, I can hit our A station to be listed! ! ! !

\

\

40

\

Q: The performance of the iPhone processor is doubling every year. Will it be able to catch up or even surpass that of a desktop computer soon?

\

A: When I was young, I always felt that in two years I would be as old as my brother who was two years older.

\

\

41

\

Q: What is the minimal benefit that Zhihu brings to you?

\

A: It s not a sin to kill time.

\

\

42

\

Q: What are the anti-human scientific inventions or designs?

\

A: The computer is not connected to the Internet. After the diagnosis, it prompts me to connect to the Internet.

\

\

43

\

Q: Why don't designers want to be called artists?

\

A: As long as the salary is high, you can call me aunt.

\

\

44

\

Q: Why do some people think that NetEase Cloud Music is the conscience of the industry?

\

A: One day I suddenly pushed me a message saying that the lyrics I wanted were found

\

\

45

\

Q: Why is there no drone self-destructing attack weapon? Has it been used by terrorists?

\

A: Are you talking about missiles?

\

\

46

\

Q: Since thoughts are mine, why sometimes I can't control my negative emotions?

\

A: The operating system will not allow users to access, modify and delete core system files, because this will damage the system and cause abnormal operation.

\

\

47

\

Q: Although Lu Xun is very good, does he make up the number among the top ten writers in the world?

\

A: Why do writers pay for the illiterate rankings?

\

\

48

\

Q: Which human technologies have been close to the bottleneck and haven't had any major breakthroughs for a long time?

\

A: Boil water

\

\

49

\

Q: How do you think some people s preference for downloading software to the official website?

\

A: Classmate, you haven't been in Baidu Family Bucket, right?

\

\

50

\

Q: Why do many people buy laptops to play games instead of desktop computers with better performance?

\

A: Because I can't afford a house. . .

\

\

51

\

Q: How shocking did you listen to headphones for the first time?

\

A: Listening to headphones for the first time will not give people much shock, but when you switch back to ordinary headphones, the shock will come.

\

\

52

\

Q: Does Chrome really consume electricity?

\

A: It doesn't cost electricity. I'm using Chrome now. After using it for so long, my laptop battery is still 50%. I estimate

\

\

53

\

Q: What is your experience after installing Windows on your MacBook?

\

A: Suddenly I have weakness and lost my armor.

\

\

54

\

Q: What is the experience of using Apple products for all relevant products at home?

\

A: The whole family rang after a call

\

\

55

\

Q: Why don't you buy iPhone X?

\

A: The contradiction between the growing need for a better life and the reality of poverty

\

\

56

\

Q: Why are some people willing to spend thousands of dollars to buy iPhone, but they are not willing to spend tens of dollars to buy genuine iPhone software and games?

\

A: Because they can't download iphone

\

\

57

\

Q: Is there any app that has a particularly amazing name?

\

A: Water meter assistant... is for checking express delivery...

\

\

58

\

Q: Why do you want to buy a mobile hard drive?

\

A: Once the conditions are good, you should give your women a more comfortable stay.

\

\

59

\

Q: How to remotely shut down PC with iPad?

\

A: Aim at the pc power button and throw it over

\

\

60

\

Q: How do you evaluate the Apple conference on September 7, 2016?

\

A: For the new MacBook Pro, I watched three conferences in half a year...

\

\

61

\

Q: How to evaluate Internet Explorer?

\

A: Download the browsers of other browsers
-----one year later-----
IE8 and below are bad, and I want to cry at the front end.

\

\

62

\

Q: My parents asked me to save money to buy a house, but I want to buy an Apple computer, what should I do?

\

A: If you can really save a house of 500,000 yuan in 3 years, do you need to buy a computer for 17,000, brother?

\

\

63

\

Q: What kind of junk mobile phone software?

\

A: SMS blocking software! After intercepting, it tells you that it intercepted a text message. I believe that 99% of people will click in to see the intercepted text messages!

\

\

64

\

Q: After a complete PPT is completed, what is the most headache for you?

\

A: How to hide your strength from the leader.

\

\

65

\

Q: What can Vim do but Emacs can't?

\

A: Help the poor children in Uganda...

\

\

66

\

Q: Why do Apple users choose Apple?

\

A: Because users who don't use Apple are not Apple users.

\

\

67

\

Q: What are the classic rumors in the computer world?

\

A: Windows is looking for a solution online.

\

\

68

\

Q: Will the wired mouse be replaced by a wireless mouse?

\

A: I think wired mice will not be replaced in Internet cafes

\

\

69

\

Q: What are the classic rumors in the computer world?

\

A: I have read and agree to the terms

\

\

70

\

Q: What mantras do computer students have?

\

A: It runs fine on my computer...

\

\

71

\

Q: What do you think about Baidu's official blog publicly refuting rumors about Robin Li's family affairs?

\

A:

"The Chinese are not that sensitive to privacy and are willing to trade privacy for convenience."

Li Yanhong

\

\

72

\

Q: How can I chat when I meet Jack Ma on the plane?

\

A: Hello Jack, my name is Jackson.

\

\

73

\

Q: How do you understand Jack Ma's saying that the house will be like a green onion after eight years?

\

A:

Buy green onions, the price of green onions is going up! ! !

\

\

74

\

Q: How do you understand Ma Yun's saying "Killing the landlord does not mean you can get rich"?

\

A: He meant "Don't kill me"

\

\

75

\

Q: How do you think that after the Wei Zexi incident passed, Baidu quietly dimmed the color of the ad prompts that promised to rectify?

\

A: Please don't hack Baidu. I am doing front-end development. It has been a long time, and the CSS of the web page has faded.

\

Follow the public account data science club backstage reply " knowledge " below to get the complete code of this article

\

As a decentralized global technology community, the Python Chinese community has the vision to become a spiritual tribe of 200,000 Python Chinese developers around the world. It currently covers major mainstream media and collaboration platforms, and is closely related to Alibaba, Tencent, Baidu, Microsoft, Amazon, and open source. China, CSDN and other industry well-known companies and technology communities have established extensive contacts, with tens of thousands of registered members from more than a dozen countries and regions, members from the Ministry of Public Security, the Ministry of Industry and Information Technology, Tsinghua University, Peking University, Beijing University of Posts and Telecommunications, and the People of China Government agencies, scientific research institutions, financial institutions, and well-known companies at home and abroad represented by banks, Chinese Academy of Sciences, CICC, Huawei, BAT, Google, and Microsoft, have attracted nearly 200,000 developers on the entire platform.

Reply "internal push" at the bottom of the official account of the Python Chinese community

Get a list of technical positions pushed within a week

******** Click below**** to read the original text and become a **** community member for free