Commit Graph

83 Commits (375d9a2a3cf9c023ac6ac832d98f1b456fb9e2ce)

Author SHA1 Message Date
phiresky 375d9a2a3c
Persistent, performant, reliable federation queue (#3605)
* persistent activity queue

* fixes

* fixes

* make federation workers function callable from outside

* log federation instances

* dead instance detection not needed here

* taplo fmt

* split federate bin/lib

* minor fix

* better logging

* log

* create struct to hold cancellable task for readability

* use boxfuture for readability

* reset submodule

* fix

* fix lint

* swap

* remove json column, use separate array columns instead

* some review comments

* make worker a struct for readability

* minor readability

* add local filter to community follower view

* remove separate lemmy_federate entry point

* fix remaining duration

* address review comments mostly

* fix lint

* upgrade actitypub-fed to simpler interface

* fix sql format

* increase delays a bit

* fixes after merge

* remove selectable

* fix instance selectable

* add comment

* start federation based on latest id at the time

* rename federate process args

* dead instances in one query

* filter follow+report activities by local

* remove synchronous federation

remove activity sender queue

* lint

* fix federation tests by waiting for results to change

* fix fed test

* fix comment report

* wait some more

* Apply suggestions from code review

Co-authored-by: SorteKanin <sortekanin@gmail.com>

* fix most remaining tests

* wait until private messages

* fix community tests

* fix community tests

* move arg parse

* use instance_id instead of domain in federation_queue_state table

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: SorteKanin <sortekanin@gmail.com>
8 months ago
Dessalines 9785b20843
Adding a scaled sort, to boost smaller communities. (#3907)
* Adding a scaled sort, to boost smaller communities.

- Previously referred to as *best* .
- Fixes #3622

* Fixing scheduled task update.

* Converting hot_rank integers to floats.

* Altering hot_rank psql function to default to zero after a week.

* Setting scaled_rank to zero, where hot_rank is zero.

* Adding image_upload table.
9 months ago
Nutomic 71d61138bc
Replace ammonia lib with manual html escaping (fixes #3774) (#3938)
* Replace ammonia lib with manual html escaping (fixes #3774)

* prettier

* clippy

* remove sanitize unit test

* fix tests

* fix schema
9 months ago
Anon fe3ebea95a
Add logging for pictrs uploads (#3927)
* Add logging for pictrs uploads

* cleanup
9 months ago
Dessalines 56e26fc3d4
Adding a post_view mode. Fixes #3730 (#3731)
* Adding a post_view mode. Fixes #3730

* Fix test.

* Addressing PR comments.

* Adding a post_view mode. Fixes #3730

* Fix test.

* Addressing PR comments.

* Fixing column order.

* Fix default Ok returns.

* Removing return Err(... where feasible.
9 months ago
phiresky 514f2222e0
Fix time zone handling (#3496)
* convert naive time to utc time

* compounding fixes

* cargo fmt

* fix the rest

* fix down migration

* fix migrations

* fix after merge

* clippy fix

* ap-fed 0.5.0

---------

Co-authored-by: Nutomic <me@nutomic.com>
9 months ago
Nutomic 6047257bfc
Move admin flag from person to local_user (fixes #3060) (#3403)
* Move admin flag from person to local_user (fixes #3060)

The person table is for federated data, but admin flag can only
apply to local users. Thats why it really belongs in the local_user
table. This will also prevent the federation code from accidentally
overwriting the admin flag

* fmt

* try to fix api tests

* lint

* fix person view

* ci

* ci

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
9 months ago
Nutomic 2d0f77af59
Dont use sha hash for password reset token (fixes #3491) (#3795) 10 months ago
Pavlos Smith c890797b37
Add controversial ranking (#3205)
* Added controversy rank property to posts and comments, and ability to sort by it

* Triggers instead of schedules tasks, integer -> double, TODO: comments don't seem to get updated with floats, divide SortTypes

* Created PersonSortType

* PersonSortType::MostComments case

* Removed unused PartialOrd trait

* Added new person sort type mappings

* SortType -> PersonSortType

* fixes

* cargo fmt

* fixes after merge with main

* Fixed bug in controversy rank trigger, removed TopX sorts from PersonSortType and added CommentScore instead

* Uncovered enum case

* clippy

* reset translation changes

* translations

* translations

* Added additional hot ordering on controversial posts and comments

* featured local and featured community added to controversy rank index, additional order_by removed (?), added post_score and post_count to PersonSortType

* Woodpecker rerun

* cargo fmt

* woodpecker rerun

* fixed controversy_rank order

* fix

* Readded migration as latest, removed second update statement for setting controversy rank
10 months ago
Domenic Horner f9351b6512
UI Settings - Blur NSFW & Auto Expand (#3377)
* add new flag to api

* add new ui settings for local user

* remove extraneous def

* add props to application reg.

* fix clippy updated these

* re-order db schema entries

* remove dupe

* update lemmy sdk

* update lemmy js client

---------

Co-authored-by: Nutomic <me@nutomic.com>
10 months ago
Sander Saarend b511c2e6cb
Denormalize community_id into post_aggregates for a 1000x speed-up when loading posts (#3653)
* Denormalize community_id into post_aggregates for a 1000x speed-up when loading posts

* Remove unused index

* Add creator_id to post_aggregates

* Use post_aggregates as main table for PostQuery

* Make post_aggregates the main table for PostView

* Reformat SQL
10 months ago
Nutomic e9e76549a8
Split activity table into sent and received parts (fixes #3103) (#3583)
* Split activity table into sent and received parts (fixes #3103)

The received activities are only stored in order to avoid processing
the same incoming activity multiple times. For this purpose it is
completely unnecessary to store the data. So we can split the
table into sent_activity and received_activity parts, where
only sent_activity table needs to store activity data. This should
reduce storage use significantly.

Also reduces activity storage duration to three months, we can reduce
this further if necessary.

Additionally the id columns of activity tables are removed because
they are completely unused and risk overflowing (fixes #3560).

* address review

* move insert_received_activity() methods to verify handlers

* remove unnecessary conflict line

* clippy

* use on conflict, add tests
10 months ago
Simon Bordeyne 9b5e765364
Add infinite scroll user option (#3572) 10 months ago
Simon Bordeyne b8ee9315bc
Add Open links in new tab setting (#3318)
* Add Open links in new tab setting

* reorder because it fixes tests ?
11 months ago
TKilFree 2aef6a5a33
feat: re-added captcha checks (#3289) 11 months ago
Nutomic ddfa112e0b
Allow wildcard imports in schema.rs (#3293)
Dont mess with auto-generated code, this avoids problems with
clippy after running diesel commands
11 months ago
Scott c5886404ef
Update DB local_user.theme type to text (#3266)
* Update local_user.theme type to text

* fix default value

* Undo auto generate changes
11 months ago
cetra3 d7da911a48
Remove `actix_rt` & use standard tokio spawn (#3158)
* Remove `actix_rt` & use standard tokio spawn

* Adjust rust log back down

* Format correctly

* Update cargo lock

* Add DB settings

* Change name and update to latest rev

* Clean up formatting changes

* Move `worker_count` and `worker_retry_count` to settings

* Update defaults

* Use `0.4.4` instead of git branch
11 months ago
Dessalines 1f1c0ad17a
Revert "feat: re-added captcha checks (#3249)" (#3288)
This reverts commit 8a086c8240.
11 months ago
TKilFree 8a086c8240
feat: re-added captcha checks (#3249) 11 months ago
Dessalines 8cb5939f50
Adding hot_rank columns in place of function sorting. (#2952)
* Adding hot_rank columns in place of function sorting.

- Creates hot_rank columns for post, comment, and community.
- Fixes #2932

* Updating all hot ranks on startup.

* Fixing post.url migration.

* Removing update_instance_software from startup.

* Adding post_rank query
12 months ago
Nutomic 2423b89ced
Activitypub breaking changes for Lemmy 0.18 (#2705)
* Remove deprecated field `group.moderators`

* Drop column local_site.federation_debug

* Remove backwards compatibility code for feature/lock post

https://github.com/LemmyNet/lemmy/pull/2716

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
12 months ago
Carlrs 24323e17b2
update diesel to 2.1.0 and diesel-async to 0.3.0 (issue #2882) (#2895)
* update diesel to 2.1.0 and diesel-async to 0.3.0 (issue #2882)

* get rid of deprecated RawValue type alias usage, explicitly import diesel types instead of using wildcard import

* Update diesel-derive-enum and diesel-derive-newtype libraries

* Revert manual changes to schema.rs

* Update to newer diesel-cli image in CI, run fix-clippy before schema comparison, update schema to one generated with diesel-cli 2.1.0 and clippyfixed

* Remove wildcard import lint, fix diesel schema pipeline

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
12 months ago
Dessalines d8722b6e91
Adding diesel enums for SortType and ListingType (#2808)
* Adding diesel enums for SortType and ListingType

- Uses diesel-derive-enum.
- Adds diesel.toml , so we can again use the auto-generated schema.rs
- Fixes a lot of DB null issues and column ordering issues.
- Fixes #1136
- Also replaces RegistrationMode boilerplate.

* Fixing unit tests 1.

* Remove comment line.

* Before patch.

* Before again.

* Using patch file to fix diesel_ltree issue with diesel.toml

* Adding some yalc ignores

* Fixing RegistrationMode enums

* Adding woodpecker diesel schema check.

* Try adding openssl 1.

* Try using diesel-cli image 1

* Try using diesel-cli image 2

* Try using diesel-cli image 3

* Try using diesel-cli image 4

* Try using diesel-cli image 5

* Try using diesel-cli image 6

* Try using diesel-cli image 7

* Try using diesel-cli image 8

* Try using diesel-cli image 9

* Try using diesel-cli image 10

* Try using diesel-cli image 11

* Try using diesel-cli image 12

* Try using diesel-cli image 13
1 year ago
Anon 6bc49bdd70
Add Custom Emojis Support (#2616)
* Add Custom Emojis

* Modify index
1 year ago
Dessalines 1dba94c9cb
Adding TOTP / 2FA to lemmy (#2741)
* Combine prod and dev docker setups using build-arg

- Fixes #2603

* Dont use cache for release build.

* Adding 2FA / TOTP support.

- Fixes #2363

* Changed name to totp_2fa for clarity.

* Switch to sha256 for totp.
1 year ago
Nutomic 62663a9f2e
Implement separate mod activities for feature, lock post (#2716)
* Implement separate mod activities for feature, lock post

Also includes collection for featured posts. Later we also need
to do the same for Comment.distinguished

* some changes

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
1 year ago
Dessalines 3735c6fabf
Adding instance software and version. Fixes #2222 (#2733)
* Adding instance software and version. Fixes #2222

* Fix clippy.

* Fix clippy 2

* Fixing some more issues.
1 year ago
Dessalines ddb6268164
Adding a site option to email admins for new reports. (#2730)
- Fixes #2551
1 year ago
Nutomic ceff2ec686
Use enum for registration mode setting (#2604)
* Use enum for registration mode setting

* fix tests
1 year ago
Anon 9dfd819691
Add support for Featured Posts (#2585)
* Add support for Featured Posts

* Fix rebase

* More fixes
1 year ago
Nutomic d20d2b9218
Implement federated user following (fixes #752) (#2577)
* Implement federated user following (fixes #752)

* rewrite send_activity_in_community and add docs, remove default for column pending

* improve migration

* replace null values in db migration
2 years ago
Nutomic 872c60a013
Remove federation settings, rely on sensible defaults instead (#2574)
This affects federation_strict_allowlist and federation_http_fetch_retry_limit
2 years ago
Anon c27d2a5687
Add support for Taglines (#2548)
* Add support for Taglines

* recreate migration

* Update content column -> Text

* Optimize replace function

* Change taglines to Option

* Move Ok() call

* Move taglines to GetSiteResponse

* Fix logic

Co-authored-by: layla <layla@chapo.dev>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2 years ago
Dessalines 235cc8b228
Moving settings to Database. (#2492)
* Moving settings to Database.

- Moves many settings into the database. Fixes #2285
- Adds a local_site and instance table. Fixes #2365 . Fixes #2368
- Separates SQL update an insert forms, to avoid runtime errors.
- Adds TypedBuilder to all the SQL forms, instead of default.

* Fix weird clippy issue.

* Removing extra lines.

* Some fixes from suggestions.

* Fixing apub tests.

* Using instance creation helper function.

* Move forms to their own line.

* Trying to fix local_site_data, still broken.

* Fixing federation tests.

* Trying to fix check features 1.

* Addressing PR comments.

* Adding check_apub to all verify functions.
2 years ago
Nutomic 2ef0f8f5f8
implement language tags for site/community in db and api (#2434)
* implement language tags for site/community in db and api

* add api checks for valid languages

* during db migration, update existing users, sites, communities to have all languages enabled

* init new users/communities with site languages (not all languages)

* federate site/community languages

* fix tests

* when updating site languages, limit community languages to this subset

also, when making a new post and subset of user lang, community lang
contains only one item, use that as post lang

* add tests for actor_language db functions

* include language list in siteview/communityview

* Fix some of the review comments

* Some more review changes

* Add todo about boxed query

* Add default_post_language to GetCommunityResponse
2 years ago
Dessalines ae95f5928e
Adding email admins for new applications. Fixes #2271 (#2390)
* Adding email admins for new applications. Fixes #2271

* Fix error.

Co-authored-by: Nutomic <me@nutomic.com>
2 years ago
Dessalines 0aeb78b8f3
Showing # of unread comments for posts. Fixes #2134 (#2393)
* Showing # of unread comments for posts. Fixes #2134

* Fix lint.

* Forgot to remove comment list update.

* Fix clippy
2 years ago
Dessalines c9f1407429
Diesel 2.0.0 upgrade (#2452)
* Initial commit to bump diesel to 2.0.0-rc.0 and see what happens

* Add chrono feature from diesel

* db_schema crate is close to building?

* Upgrade diesel-derive-newtype

* Mostly modifying references to connections to be mutable ones; also used
new way to do migrations as suggested by the migration guide; a lot more
compiles now, though I can't figure out this tricky ToSql issue at the
moment

* Running clippy --fix

* Trying to fix drone clippy 1

* Fix clippy

* Upgrade clux-musl

* Trying to fix drone clippy 2

* Trying to fix drone clippy 3

* Trying to fix drone clippy 5

* Adding diesel table aliases, removing sql view hack. Fixes #2101

Co-authored-by: Steven Chu <stevenc1@gmail.com>
Co-authored-by: Nutomic <me@nutomic.com>
2 years ago
Nutomic 004efd5d94
Implement reports for private messages (#2433)
* Implement reports for private messages

* finish private message report view + test

* implement api for pm reports

* merge list report api calls into one, move report count to site

* fix compile error

* Revert "merge list report api calls into one, move report count to site"

This reverts commit 3bf3b06a705c6bcf2bf20d07e2819b81298790f3.

* add websocket messages for pm report created/resolved

* remove private_message_report_view

* add joinable private_message_report -> person_alias_1

* Address review comments
2 years ago
Felix Ableitner 3d0d8796ad Add language tags for comments 2 years ago
Nutomic e4a49b6eab
Tag posts and comments with language (fixes #440) (#2269)
* Tag posts and comments with language (fixes #440)

* Untangle PostView tests

* Implement test for PostView language query

* Store languages directly in database

* finish moving languages into db, it compiles

* update post_view

* serde skip Language.id field

* add local_user_language table, other changes suggested in review

* add code for local_user_discussion_language_view

* Remove unnecessary clones in db view converteres

* Fixing up some table and join issues.

* Clearing the current languages.

* Fix formatting.

* update user languages in single transaction

* proper test for user language queries

* Some fixes for all / missing user languages. (#2404)

* Some fixes for all / missing user languages.

* Adding back in transaction.

* fix test

Co-authored-by: Dessalines <tyhou13@gmx.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2 years ago
Dessalines 870abf8442
Adding distinguish comment. Fixes #2002 (#2391)
* Adding distinguish comment. Fixes #2002

* Adding lemmy:distinguished
2 years ago
Anon 21455d6b73
Add Modlog Filters (#2313)
* Fix crash running locally on windows.

* Add support for filtering mod logs

* Refactor cleanup

* Clippy fix

* Condense match statements

* Clippy fix 2
2 years ago
Dessalines 9c3efe32e7
First pass at adding comment trees. (#2362)
* First pass at adding comment trees.

- Extracted comment replies into its own table.
- Added ltree column to comment
- Added parent_id param to GetComments to fetch a tree branch
- No paging / limiting yet

* Adding child_count to comment_aggregates.

* Adding parent comment update counts

* Fix unit tests.

* Comment tree paging mostly done.

* Fix clippy

* Fix drone tests wrong postgres version.

* Fix unit tests.

* Add back in delete in unit test.

* Add postgres upgrade script.

* Fixing some PR comments.

* Move update ltree into Comment::create

* Updating based on comments.

* Fix send soft fail.
2 years ago
Dessalines 4e12e25c59
Adding admin purging of DB items and pictures. #904 #1331 (#1809)
* First pass at adding admin purge. #904 #1331

* Breaking out purge into 4 tables for the 4 purgeable types.

* Using CommunitySafe instead in view

* Fix db_schema features flags.

* Attempting to pass API key.

* Adding pictrs image purging

- Added pictrs_config block, for API_KEY
- Clear out image columns after purging

* Remove the remove_images field from a few of the purge API calls.

* Fix some suggestions by @nutomic.

* Add separate pictrs reqwest client.

* Update defaults.hjson

Co-authored-by: Nutomic <me@nutomic.com>
2 years ago
Nutomic 339eab01fd
Embed Peertube videos (#2261)
* Use og:video attribute for embeds, change Post.embed_html to embed_url

* fix clippy
2 years ago
Dessalines eafb3869f2
Revert "Add pending, and change use specific API response for FollowCommunity…" (#2283) 2 years ago
Dessalines d4ee171b08 Making community_follower.pending column not null. 2 years ago
Nutomic 9ef0efa420
Add legal information (fixes #721) (#2273) 2 years ago